# Language assessment in multilingual settings

Innovative practices across formal and informal environments

Edited by Eva Rodríguez-González Rosita L. Rivera

Current Issues in Bilingualism 2

### Current Issues in Bilingualism

Editors: Andrea C. Schalley (Karlstad University, Sweden) [editor-in-chief], M Carmen Parafita Couto (Leiden University, Netherlands) [editor-in-chief], Susana Eisenchlas (Griffith University, Australia), Galina Putjata (Europa-Universität Flensburg, Germany), Jorge Valdés Kroff (University of Florida, USA).

### In this series:


# Language assessment in multilingual settings

Innovative practices across formal and informal environments

Edited by

Eva Rodríguez-González

Rosita L. Rivera

Eva Rodríguez-González & Rosita L. Rivera (eds.). 2022. *Language assessment in multilingual settings: Innovative practices across formal and informal environments* (Current Issues in Bilingualism 2). Berlin: Language Science Press.

This title can be downloaded at: http://langsci-press.org/catalog/book/333 © 2022, the authors Published under the Creative Commons Attribution 4.0 Licence (CC BY 4.0): http://creativecommons.org/licenses/by/4.0/ ISBN: 978-3-96110-344-7 (Digital) 978-3-98554-040-2 (Hardcover)

ISSN (print): 2747-9919 ISSN (electronic): 2747-9927 DOI: 10.5281/zenodo.6535786 Source code available from www.github.com/langsci/333 Errata: paperhive.org/documents/remote?type=langsci&id=333

Cover and concept of design: Ulrike Harbort Typesetting: Sebastian Nordhoff, Yanru Lu, Eduard Lukasiewicz, Ivo Boers, Miriam Romaneli Proofreading: Alina Bachmann, Alexia Fawcett, Amir Ghorbanpour, Annika Schiefner, Bojana Bašaragin, Caroline Rossi, Francesco Screti, Jenny Jaffe, Jeroen van de Weijer, Marten Stelling, Jean Nitzke, Plinio Barbosa, Wilson Lui Fonts: Libertinus, Arimo, DejaVu Sans Mono Typesetting software: XƎLATEX

Language Science Press xHain Grünberger Str. 16 10243 Berlin, Germany http://langsci-press.org

Storage and cataloguing done by FU Berlin

## **Contents**


### Contents


## **Praise**

The contributors in this volume are scholars from different disciplines and contexts in Higher Education. They have created and proposed multiple lower-stakes assignments and accommodated learning by being flexible and open without assuming that learners know how to do specific tasks already. Each chapter provides different examples on justice, equity, diversity, and inclusion (JEDI) assessment practices based on observation, examination and integrative notions of diverse language scenarios.

Ana Celia Zentella Professor Emerita Department of Ethnic Studies University of California, San Diego

## **Acknowledgments**

We are thankful to a number of people who helped us realize this project. Above all, we thank the individual authors who contributed their research, time, and professionalism to the present volume on alternative language assessments. We also appreciate the continuous support offered by our series editors, Susana Eisenchlas and Andrea Schalley. Some of the production of the present volume took place in the midst of the COVID-19 pandemic, making things more challenging for authors and reviewers. Patience, resilience and commitment prevailed and we are indebted to wonderful colleagues who participated in this project.

The quality of the finished product presented in the research of the following chapters was also made possible by a vigorous peer review process, and we thank our reviewers from the fields of Scholarship of Teaching and Learning, Assessment and Applied Linguistics for their professional dedication and their expertise in providing feedback to our authors. They include:


## **Foreword**

Marleen Haboud Bumachara,b

<sup>a</sup>Pontificia Universidad Católica del Ecuador <sup>b</sup>Ruhr-Universität Bochum

*Language assessment in multilingual settings: Innovative practices across formal and informal environments* edited by Eva Rodríguez-González and Rosita L. Rivera is a very important contribution, not only for academics interested in a quality educational system that erases social inequities, but above all, for students, second language learners, heritage speakers, or multilingual and multicultural speakers who struggle to adjust to a homogeneous monocultural system, divorced from their reality/social context characterized by diversity. In this sense, this book strives to join forces to build, from school and beyond, a more conscious and fair society that understands and accepts difference, not as a barrier, but as a source of wealth as an individual and collective asset.

Each chapter draws our attention to the urge to create awareness towards the student body heterogeneity and, therefore, the need to rethink and re-create the curriculum contents, the teaching-learning methodologies, as well as the urgency of leaving behind assessment practices that look at/conceive students as homogeneous entities, to instead move on towards developing strategies that take into account their particularities, abilities, needs and expectations.

This book focuses on heritage speakers who look forward to rediscovering the language of their parents, the wisdom of their ancestors, and to proudly reinforce their identity. It also reminds us of the important role we have when assessing students from different linguistic and sociocultural backgrounds, or when we act as cultural interpreters of speakers of less prestigious languages who often confront harsh living conditions.

The varied topics discussed in each chapter bring us closer to multiple territories, languages, cultures and identities, with a broad perspective and alternative paths that move us to delve into diversity with creativity and respect, and with the conviction that multilingualism must be a tool to glimpse at social justice. Indeed, each one of the authors opens up a range of strategies and possibilities for

Marleen Haboud Bumachar. 2022. Foreword. In Eva Rodríguez-González & Rosita L. Rivera (eds.), *Language assessment in multilingual settings: Innovative practices across formal and informal environments*, vii–viii. Berlin: Language Science Press. DOI: 10. 5281/zenodo.7389662

us to contribute towards building a more equitable society, based on respecting the difference.

Anyone who reads this book will deeply feel the urgency to deeply scrutinize their own autobiography in order to have a better understanding of the self and the other, and the many others, who have been made invisible in many of the classrooms. Each reader will be inspired to see diversity through new lenses, and to develop creative strategies to dialogue with the richness and the challenges fetched by diversity.

Anyone who is genuinely interested in walking hand in hand with their students, in understanding the profound meanings of every one of their words and gestures, in moving away from automated evaluation or mechanical translation towards intercultural dialogues that pursue social justice, will not be able to stop reading this book again and again. Near the end, the reader will not only be inspired, but truly committed to contributing in the search for *equity in diversity*.

## **Chapter 1**

## **Integrated approaches to language assessment in language learning: Introduction and chapter synopsis**

### Rosita L. Rivera<sup>a</sup> & Eva Rodríguez-González<sup>b</sup>

<sup>a</sup>University of Puerto Rico-Mayagüez <sup>b</sup>University of New Mexico

This introductory chapter establishes the rationale for the volume by situating assessment within the context of multilingual learners in diverse settings. This chapter also includes a brief definition of assessment terminology. We conclude this introduction by providing an overview of the chapters and contributions of scholars included in this edited volume.

Recent research in the field of applied linguistics has addressed the complex and contextual realities of multilingual language learners (Larsen-Freeman 2018, 2017; Ortega 2017). Scholars have also discussed the dynamic nature of language and the need to address the different characteristics and experiences learners bring with them to their learning context (García & Wei 2014; García 2009; Lantolf et al. 2020). There is a need for adaptability and creation of multiple ways to assess linguistic knowledge to address the needs of multilingual citizens. It is within this context that the concept of "assessment" is gaining momentum. Educational institutions are constantly dealing with the design of "reliable metrics to ensure that individuals have sufficient linguistic competence to carry out job- or school-related tasks and also to compare the language capabilities of individuals" (Menke & Malovrh 2021: 17). In recent decades, alternative views of language assessment have been proposed and advocated for language learning. This volume includes examples of these alternative views in multiple contexts to illustrate how the learning environment determines the design of assessment that suits the specific needs of the learners.

Rosita L. Rivera & Eva Rodríguez-González. 2022. Integrated approaches to language assessment in language learning: Introduction and chapter synopsis. In Eva Rodríguez-González & Rosita L. Rivera (eds.), *Language assessment in multilingual settings: Innovative practices across formal and informal environments*, 1–9. Berlin: Language Science Press. DOI: 10.5281/zenodo.6762270

Study the science of art. Study the art of science. Develop your senses – especially learn how to see. Realize that everything connects to everything else.

Leonardo da Vinci

### **1 Rationale and significance of the book**

This book grew out of our intention to provide the reader with examples of what it means to design and implement research from a context-based perspective and how this view differs from other approaches to assessment. We address the following questions related to language assessment: Is it only instructors who assess language?; What other options become available to language testing inside and outside of a language classroom setting?; How do we assess different language profiles based on language exposure and experience outside of classroom boundaries?

This volume attempts to introduce and address these questions in order to promote equitable access for assessment and initiate a conversation among scholars about inclusive practices in language assessments. Rather than universal or applicable ("it describes everyone") to everyone, we work with the notion of teaching in multiple contexts. However, each context has specific needs that the contributors in this volume address based on the linguistic experiences and realities of their specific context. Whether the student is a second language learner, a heritage language learner, a multilingual language speaker, a community member, the authors in the present volume provide examples of assessment that do not follow a single universal or standardized design but an applicable one based on the needs and context of a given community.

The contributors in this volume are scholars from different disciplines and contexts in Higher Education. They have created and proposed multiple lowerstakes assignments and accommodated learning by being flexible and open without assuming that learners know how to do specific tasks already. Each chapter provides different examples on Justice, Equity, Diversity, and Inclusion (JEDI) assessment practices based on observation, examination and integrative notions of diverse language scenarios.

This volume is relevant at this particular point in time due to the need of addressing and validating contexts in which language assessment goes beyond the standard testing and evaluation practices. It also serves to provide examples of what it means to assess learners as a grassroots movement rather than a top

### 1 Integrated approaches to language assessment in language learning

down approach. Our intent is not to privilege one approach to assessment or one context over another, but to argue that researchers and practitioners may choose what they deem valuable research theories and techniques in their particular setting. We provide a descriptive approach to research in assessment.

This volume may be of interest to researchers and practitioners in the fields of curriculum and instruction, language learning, and applied linguistics as well as those in the field of language teaching in general. In the following sections, we define and problematize key terms. We also provide an overview of the chapters in the volume.

### **2 Definition of key terms**

In order to investigate the role of assessment in language education, this section explores and defines key terminology as it relates to assessment in multilingual settings. These key terms include assessment, evaluation, and testing. We contextualize these definitions based on the studies included in this volume and language learning in diverse contexts.

### **2.1 Assessment**

Assessment goes beyond a final product and how students are able to use language. It includes formative and summative components in which learners are evaluated based on their ongoing performance. Historically, most of the research in Second Language Acquisition serves as the basis for a great body of literature in the area of language assessment. Many of these studies are based on psychology and variables that require data elicitation (Ortega 2017; Brown et al. 2017). These data are also self-reported in many instances, but they are measured. Examples of these are surveys and questionnaires measuring aptitude and attitudes as well as motivation. Foreign language testing and proficiency tests were developed as a response to the knowledge of foreign languages increasing with World War II and US involvement with Korea, and the realization that soldiers in these contexts became bilingual or multilingual (Brown et al. 2017). This generated the first proficiency exams taking into consideration language abilities, language skills, and communicative competence. Placement tests became a venue to assess language.

In this volume, we provide examples of what could also be considered an ecological approach to assessment. The ecological perspective on language learning (Larsen-Freeman 2017; Van Lier 2010; Van Lier 1997) explores relationships of

many kinds in and across settings and systems as a way to examine relations and processes between learning and the environment. An ecological approach to language assessment acknowledges that an individual is not only a measure of variables in traditional types of assessment such as aptitude, attitude, and motivation. The ecological orientation of language assessment acknowledges that the individual is the result of multiple factors and interactions with the historical, sociocultural, and sociopolitical context. This involves the study of language anxiety, emotions, language ideologies, language policies, and power relations. In the formal environment of the classroom, individuals carry with them all these experiences, values, and beliefs. These experiences challenge educators to design assessment practices that acknowledge these dynamics in the context of formal education. This approach focuses primarily on the quality of learning opportunities, of classroom interaction and of educational experience in general.

### **2.2 Evaluation**

Evaluation is concerned with revising curricula and programs in order to provide a better view of what works or what needs to be modified. It includes a process that may lead to changes to curriculum and to ways in which students are being taught and assessed. For evaluation to be effective, there has to be an understanding of the linguistic needs of the learners as well as the resources available to them. These include the learning environment and an inventory of resources available to practitioners in order to generate curricula adapted to the learners' needs. Language Policies as well as curriculum development are essential components of these evaluation processes and are usually the outcome. Evaluation is cyclical and necessary in order to support language learning. Research is a fundamental part of the evaluation process. The key element that helps us to understand how evaluation works in multiple settings is community involvement and sensitivity to recognize the needs of diverse communities of language learners. It is within this perspective that the exemplary studies in this volume illustrate what it means to be inclusive through language evaluation processes and how evaluation leads to curriculum design that works for a specific community.

### **2.3 Testing**

Although there has been a shift in testing and adapting assessment to more authentic situations, standardized testing is still at the core of foreign and second language instruction. Literature in the field of foreign language education (FL) and L2 learning argue that standardized testing is not a direct reflection of

### 1 Integrated approaches to language assessment in language learning

the communicative competence in multilingual contexts (Brown 2010). However, even though we continue to explore the role of standardized testing, it is still unclear how standardized testing assesses students' ability to show communicative competence and pragmatic knowledge of foreign language learners, heritage language learners, multilingual learners in a variety of contexts.

Standardized testing is the direct result of a positivist epistemological stance in which the aim is to form generalizations based on results. Reliability is considered in terms of numbers and exams are designed to measure knowledge from a prescriptivist perspective (Ortega 2017; Malovrh & Menke 2021). This dates back to IQ testing and the bell curve in which results are measured and learners are compared based on their results.

Testing in diverse and multilingual settings raises the question of validity when designing diverse types of assessment. As we introduce the authors and their contributions to this volume in the following section, we also discuss how testing is conceptualized in the particular contexts of the studies represented in the chapters.

### **3 Overview of the chapters**

The challenge of assessment in multilingual communities requires that educators contest more traditional and prescriptive notions of assessment to better serve their communities of learners. Some of these diverse contexts in higher education include different languages and different learner profiles. We feel it is timely to place the local contexts in assessment at the forefront of research and practice in multilingualism and linguistics studies within the landscape of a diverse perspective. As such, the different chapters of the volume serve as a collection of studies on the sociological, pedagogical and linguistic characteristics of language learning and assessment in different institutions and learning environments. Each chapter addresses a variety of different approaches and methods in language assessment, ranging from self-efficacy assessment tools such as Can-Do assessments to integrative approaches that include multiple sources of data beyond language proficiency. One of the overarching questions that guide the volume, related to the pedagogical challenges and needs heritage language learners face in terms of assessment, is examined in multiple chapters of the volume. The results reported in the chapters are a starting point for discussion about curriculum design and strategies to monitor and assess language growth of a variety of language speakers and learners within the classroom and beyond. The role of the local communities is also mentioned and discussed in most of the chapters

as a way to organically integrate and connect instruction, language proficiency and language development.

In Chapter 2, Gregory Thompson proposes three alternative assessments to better understand students' abilities in the target language (Spanish) as well as their overall proficiency in Higher Education classroom-based contexts, namely, assessment in community-based language learning, Integrated Performance Assessments (IPAs), and student portfolios to document language growth and development. Thompson makes a strong recommendation regarding the importance of assessing for learning (vs. "of" learning). In this regard, he advocates for curricular implementation of formative and ongoing assessments that allow both the instructor and the learner to make a more explicit connection between what is taught and what is learned and assessed through constant and detailed feedback.

In Chapter 3, Eva Rodríguez-González, María de los Angeles Giráldez-Elizo and Sarah Schulman explore student self-perceptions for curricular alignment across different language Programs of Spanish language learning in Higher Education. The authors have used an assessment tool, a Can-Do Statements survey that highlights learners' perspective and individual reflection of learning growth and development as perceived during the duration of a specific course. Similar to Thompson's chapter, the authors also focus on Spanish language learning instruction and make recommendations for the multiple profiles of Spanish language learners in a college setting in the US, namely, Spanish second language learners and heritage learners. They also propose the implementation of Can-Do statements as both a reflection activity "for" learning and ongoing assessment.

In Chapter 4, Todd Hernández provides an overview of current research on pragmatic competence when learning second languages. It also describes the nature of the methodology used to assess second language pragmatic development in a study abroad context. Hernández identifies multiple assessment methodologies used to measure pragmatic knowledge in both informal and formal contexts. Based on results from a substantial literature review on pragmatic learning, Hernández highlights the importance of the potential of pedagogical interventions before and during a given study abroad experience to enhance pragmatic learning.

In Chapter 5, Kendra Dickinson and Glenn Martínez present the case of an integrated assessment of Healthcare Interpreting Competencies among Spanish heritage language learners. The proposed assessment includes student development and career-readiness from a variety of different perspectives. The authors share the results from Spanish heritage language speakers' participation from the IMPACT program (Interpreters for the Medical Profession through Articulated Curriculum and Training). The results indicate that language proficiency,

language attitudes and career decision self-efficacy were key factors that were positively affected by the participation in the program. They also highlight the implementation of the Career Decision Self-Efficacy Scale Form as a useful measure of career-readiness and assessment tool to identify areas of need and specific intervention.

In Chapter 6, Gláucia Silva analyzes assessment in community-based heritage language schools and the case of Brazilian Portuguese in the United States and calls for an inclusive approach to assessment that includes families, learners and educators. Silva shares the results from a survey distributed to Brazilian Portuguese language teachers and school administrators in the U.S. The profile of learners in her study includes heritage of Portuguese and younger populations from previous chapters of the volume. The answers provided on the survey call for consideration of conversations to parents regarding the use of Portuguese at home as a key factor for linguistic and cultural growth. In addition to the importance of including family when maintaining and developing Portuguese as a heritage language, Silva recommends the consideration of organizing activities related to Brazilian traditions as a way to make explicit connections between the school and the community. She also addresses the complexity behind placement and provides multiple suggestions regarding assessing children's progress in Brazilian Portuguese in community-based heritage language programs.

In Chapter 7, Rosa Vallejos, Fernando García and Haydée Rosales Alvarado discuss indigenous languages in Higher education in case studies from the Amazon of Peru. They focus on Kukama and Kichwa as heritage languages. Instruments for assessment include videos, picture cards and a survey addressing the nature of social and cultural factors that play a role in learners' attitudes towards dialectal and generational varieties, self-assessment and motivation of heritage language abilities and language choices. The use of multiple assessment tools allows the authors to connect data and address assessment from a holistic point of view and as an inclusive practice that monitors individual linguistic and cultural growth and development. Given the degree of endangerment of the languages under study, the authors identify as an important finding of this chapter the fact that endangered languages can be relearned in well-structured instructional settings.

In Chapter 8, Rosita L. Rivera and Eva Rodríguez-González provide a concluding essay that discusses context-based approaches to assessment as an eclectic approach that requires a robust knowledge and understanding of the linguistic diversity of language learners. This chapter makes the case for a linguistic sensitivity and pedagogical training beyond prescriptive methodologies. It also calls for a more inclusive approach to the design and implementation of assessment

in higher education. The chapter ends by drawing on the findings from previous chapters and posing the argument that an organic approach to assessment in education and linguistic diversity in multilingual contexts deserves a place in language research. Pedagogical implications and challenges posed by the need to assess in times of crisis and the impact of technology on assessment during recent events are also discussed. Suggestions for further research in assessment in multilingual contexts are also made based on these implications.

### **References**


## **Chapter 2**

## **Current trends in language assessment: Using alternative assessments in the language classroom**

### Gregory L. Thompson

Brigham Young University

While standardized assessments play an important role in understanding and measuring overall second language proficiency, instructors are often looking for additional ways to measure student proficiency in a way that better reflects the classroom practices and interactions of their students. This chapter looks at several alternative assessments to better understand students' abilities in the target language as well as their overall proficiency. The first section of this chapter looks at the use of community-based language learning (CBLL) as a means to take students out of the classroom and provide them with opportunities to use the target language in meaningful context while serving within the community. The second section focuses on the use of Integrated Performance Assessments (IPAs) as an alternative to traditional assessments. IPAs provide students with the opportunity to include the three modes of communication: interpretive, interpersonal, and presentational in their assessments. IPAs allow students to better demonstrate their overall learning across all of the language modalities as well. The third section of this chapter analyzes the use of portfolios in the language classroom as an alternative to traditional assessments. Portfolios have been shown to offer students not only a way to gauge their progress and development but also a chance to reflect on their learning and plan for future language development. Finally, this chapter offers some concluding thoughts as well as the inclusion of an appendix with additional resources for developing these types of assessments and implementing them in the language classroom.

Gregory L. Thompson. 2022. Current trends in language assessment: Using alternative assessments in the language classroom. In Eva Rodríguez-González & Rosita L. Rivera (eds.), *Language assessment in multilingual settings: Innovative practices across formal and informal environments*, 11–33. Berlin: Language Science Press. DOI: 10 . 5281/zenodo.6762272

### Gregory L. Thompson

### **1 Introduction**

Foreign language assessment continues to be a key factor in understanding both student performance and proficiency. As language programs continue to develop and evolve, there has been a push to increase the accountability of such programs through improved assessment (Bernhardt 2006; Norris 2006). The increase in accountability is especially important in today's current environment in the Humanities where language programs are shrinking or being eliminated (Johnson 2021). This reduction in language programs is being done in spite of the fact that the need for competent language professionals has continued to grow and expand to meet an increasingly globalized world. The problem in the United States with many language programs at the college level is that despite valiant efforts, many students are graduating at the Intermediate High level or lower after finishing their degrees (Rifkin 2005; Magnan 1986; Swender 2003; Tschirner 2016). While Intermediate High level students do have some ability, the American Council on the Teaching of Foreign Languages defines this level of proficiency as being speakers who can deal with "routine tasks and social situations" and can handle "uncomplicated tasks and social situations requiring an exchange of basic information" (ACTFL 2012: 7). This is not the level of proficiency needed to function and perform as competent language professionals in a wide variety of settings.

While standardized assessments of proficiency such as the Oral Proficiency Interview (OPI), The European Language Certificates (TELC), Japanese-Language Proficiency Test (JLPT), Diplomas de Español como Lengua Extranjera (DELE), etc. continue to be used and are valuable in comparing learners across a broad range of settings, more and more instructors are looking to alternative forms of assessment and evaluation of their students that better reflect the classroom practices and procedures as well as the preparation of their students to meet the current demands of language professionals. Instructors are also looking for ways to assess a more diverse student body who make up the fabric of many language classrooms and represent a wide range of cultures as well. In addition, there has been an emphasis over the last several decades on assessments that integrate culture into the language curriculum (Bennett 1986; Byram 1997; Pedersen 2010; Schulz 2007). Byrnes et al. (2010) highlight the disconnect that exists between language and culture learning in foreign language programs at all levels and the lack of programs that develop translingual and transcultural students. Sykes (2017: 120) discusses the importance of developing a transnational languaculture "in which language and culture transcend national boundaries, are uniquely tied to individuals (not only particular languages or cultures), and de-

### 2 Current trends in language assessment

velop across a lifetime as learners move between a variety of contexts, locations, and languages".

In order to help language students move in the direction of greater proficiency and intercultural competence, different types of assessments may be needed to push students into areas where they are better able to develop these skills. Stiggins & Chappuis (2006) state that the paradigm of assessment needs to change from one of the assessment OF learning to an assessment FOR learning. They declare, "Assessment for learning happens in the classroom and involves students in every aspect of their own assessment to build their confidence and maximize their achievement" (2006: 11). Brown & Thompson (2018) highlight three challenges faced in implementing changes in the overall assessment structure of many language programs referring specifically to Spanish programs:

The current status of assessment in many collegiate Spanish programs at the course and program levels is riddled with ironies: (1) Many instructors are interested in student learning and are sure it is taking place, but are unclear how to validly demonstrate it; (2) the primary mechanism accepted by key stakeholders (e.g., administrators, donors, and parents) to demonstrate effectiveness is through valid assessment, yet many instructors resist attempts to improve it or incorporate it; and (3) persistent complaints from faculty about top-down mandates imposed by external parties such as accrediting agencies precede stiff resistance to take ownership of the process. (2018: 137–138)

This chapter will briefly discuss three types of alternative assessments that can be used as tools of assessment for learning and that can be incorporated into a wide variety of language classrooms-community-based language learning (CBLL), Integrated Performance Assessments (IPAs), and portfolios. Pierce & O'Malley (1992) define alternative assessments as methods for determining student understanding and growth, informing, and leading to changes in teaching, criterion-referenced, authentic, integrating multiple language skills, and consisting of a variety of non-traditional assessments including teacher observation, performance assessments and self-assessments (1992: 4). Tedick & Klee (1998) further describe how alternative assessments evaluate students:

Alternative assessments are not only designed and structured differently from traditional tests, but are also graded or scored differently. Student performance is evaluated on the basis of clearly defined performance indicators, criteria, or standards that emphasize students' strengths instead of highlighting their weaknesses. (1992: 3)

### Gregory L. Thompson

These assessments can help educators develop a clearer idea of the different abilities and overall learning of language students as well as address how diverse populations can benefit from thinking outside of the traditional assessment box.

### **2 Community-based language learning**

Many educators are looking for ways to involve language learners in a broader community and use their language skills towards advancing the public good while creating informed citizens and community members. Bringle et al. (2004) pose the following questions regarding the responsibility of higher education in the formation of students.


One of the ways to engage students and address these questions is through community-based learning (CBL) which falls under the broad umbrella of experiential-based learning. According to Mooney & Edwards (2001), "Communitybased learning refers to any pedagogical tool in which the community becomes a partner in the learning process" (2001: 182). Clifford & Reisinger (2019) further specify community-based learning: "Broadly speaking, community-based learning (CBL) serves as an umbrella term for activities that engage students within their communities and is often equated with service learning" (2019: 5). CBL is focused on the concept of working with community partners in a collaborative relationship. Jacoby (2015) found community engagement to be a high-impact educational practice that increases "the odds that students will invest time and effort; participate in active challenging learning experiences; experience diversity; interact with faculty and peers about substantive matters; receive more frequent feedback; and discover the relevance of their learning through real-world experiences" (2015: 11). While CBL has many similar features to other types of experiential learning such as internships, field work, volunteerism, or community service, it has certain distinct features that separates it from these other types of learning. Jacoby (2015), referring specifically to service learning, defines it as

### 2 Current trends in language assessment

"a form of experiential education in which students engage in activities that address human and community needs, together with structured opportunities for reflection designed to achieve desired learning outcomes" (2015: 1–2).

While CBL is used across a variety of academic subjects, foreign language programs are increasingly working to establish community-based language learning (CBLL) experiences that focus on the acquisition of second languages. CBLL follows the same principles as CBL but concentrates on community-based learning that is designed for the language classroom. Given the increase in programs employing CBLL, questions arise regarding not only how to successfully assess students but how CBLL relates to student learning outcomes (SLOs). Researchers have investigated using CBLL in the language classroom in order to help improve programs and students' learning. Norris (2006) writes that:

Assessments are only good insofar as their use does good, in terms of supporting educational efforts and outcomes. … Where they do not obviously support the twin goals of helping educators deliver better programs and of helping students achieve valued learning outcomes, assessments should not be used. However, in order to realize these goals fully, assessments must be used. (2006: 582)

One of the key components to CBLL is meaningful reflection which can be challenging to assess in a way that is valid and reliable. Thompson (2012) studied several advanced Spanish language classes and found that the inclusion of a CBLL component resulted in slight grade inflation since the grades were based almost entirely on simply participating in the different projects and not dependent on the quality of the CBLL. This resulted in some students raising their scores in the class by a full letter grade. He suggests that instructors need to measure "the quality of their service and reflection during the course of the class" (2012: 112) and not simply grade based on the completion of the project or a certain number of hours. There are many ideas on ways to more empirically measure student gains during CBLL, but these would be contingent on the SLOs of a particular course and program. Thompson suggests having students bring the information back into the classroom and present what they have learned as a way to assess their learning more impartially. Educators could have students do presentations before and after their CBLL experiences and compare how their cultural and linguistic knowledge has changed using detailed rubrics. Depending on the focus of the class, these presentations could be done in the target language (TL) or in the first language (L1). A conversation course focusing on speaking could have

### Gregory L. Thompson

a rubric designed to look more carefully at a student's oral expression including elements such as pronunciation, vocabulary, grammar, etc. In a conversation course, the students could initially present on the organization where they would be doing their CBLL and some of the challenges that they expect to face as well as what skills they are bringing to the experience. The post-CBLL presentation would then result in the student sharing what has been learned from the experience and the instructor would again focus on the spoken aspects of the language.

In a literature or culture class, the decision to carry out the assessment in the TL would depend on the level and the overall objectives of the course. The rubric for such a presentation would likely focus much more on the content of the presentation and the ability of the students to organize their presentations in a compelling manner of interest to the instructor and fellow students. These presentations could be less focused on overall proficiency and performance and much more on the acquisition of intercultural competence or being able to make connections between the CBLL experience and the literature being studied. Writing assignments where students compare the situation of their community partners to their own lives could also benefit them not only from language acquisition and proficiency lenses but also help them become more aware of the situation of fellow community members. These writing assignments could be very similar to the oral presentations in the sense of whether to write them in the TL or L1. Instructors would need to again consider the course objectives, proficiency of the students, and goals for the CBLL assignment. For some courses, a thoughtful, reflective composition in the L1 might benefit the students more than having lower-level students struggle to present their reflections in the TL.

Medina & Gordon (2014) investigated the role of using service learning during a language exchange between L1 English speakers and native Spanish speakers. The researchers developed a phonemic perception test that was used to measure these gains over the course of the semester. During weekly 60-minute sessions where the speakers would spend half the time practicing Spanish and the other half practicing English, adult college students were able to improve their phonemic perception. Given the use of a control group that did not use service learning, students who participated in service learning did show significant improvement over the course of the semester when compared to those who did not participate. Additionally, these researchers used a modified version of Gardner et al.'s (1997) Attitude/Motivation Test Battery and found that students who participated in service learning also had significant increases in motivation over the course of the semester when compared to those who did not participate.

### 2 Current trends in language assessment

While the study by Medina & Gordon (2014) did not include diverse learners, Lowther Pereira (2015) looked exclusively at heritage learners of Spanish who were participating in CBLL. She took a critical pedagogy approach and assessed her students' overall development through detailed self-evaluations, reflections, questionnaires, and interviews. She found that through the CBLL, the heritage language learners developed a greater "awareness of sociolinguistic and sociopolitical issues affecting local Latino communities" and were better able to construct "positive identities" (2015: 159). Salgado-Robles & Lamboy (2019) also worked with heritage learners who were pre-service teachers assigned to different schools throughout the New York City region.

The students were evaluated based off six different assignments. First, they needed to complete 30–35 hours of service in their assigned schools. They received full points for completing the hours for this assignment. Second, they completed four "checkpoint" assignments during the semester.


### Gregory L. Thompson

In the case of this class, the focus was more on the content of the experience and completing all of the components of each of the written assignments. Even though all the participants were heritage speakers of Spanish and working in Spanish-language classrooms with other Spanish-speaking students, these students were allowed to complete these activities in English since this was an education course. These same types of activities could also be developed for language courses focusing on developing the written proficiency of the students while completing these "checkpoint" assignments in the TL and having the instructor provide feedback on the language use and structure within the writing assignments.

The sixth and final assessment of the work by Salgado-Robles and Lamboy was a post-survey of their experience which was graded on the overall reflection and completion of the survey that can be found in the appendix of their article. This final survey was again written in English and mainly consisted of students selecting a number that best matched their feelings regarding the statements. The final part of the survey was composed of five open-ended questions to which the students were able to respond in English or Spanish. The researchers found through this service-learning experience that students developed a better understanding of what it means to be a teacher and what the profession entails. The participants were also better able to see the relevancy of the material from the class to their chosen profession. However, unlike the results from Lowther Pereira (2015), the results did not show any impact on the participants' view of their identities.

Both assessment by community partners and self-assessment of experiences in CBLL can serve as valuable sources for evaluating students. Regarding using community partner evaluations, Brown & Thompson (2018) state:

Although such evaluations can be problematic, given the tendency of community partners to appreciate any help that is given, these partners can be provided with targeted, confidential online surveys where they can evaluate or even rank the students who worked with them. (2018: 90)

Community partners are helpful in the evaluation of the students because they work with them and are able to help recognize their strengths and struggles as well as provide the instructor valuable with information on how to better prepare students for their CBLL experience. The authors also recommend using peer evaluations as part of an overall picture of students' performance during CBLL projects.

One way in which educators can employ a more empirically based self-reflection was outlined by Ash & Clayton (2009) through their DEAL model. While

### 2 Current trends in language assessment

the DEAL model was designed for CBL, it can be applied to the language classroom through focusing the activities and assessments on improving the students' abilities in the TL. The DEAL model consists of three different steps defined as Describe, Examine, and Articulate Learning. The first step is writing an objective and detailed description of the CBLL experience. The goal of this step is to help students see and describe their experience without the critical lens and is preparatory for more in-depth critical thinking. This could be carried out in the target language since using descriptive language can be adapted for different levels of proficiency. The instructor should develop a clear rubric for the evaluation of the students' description and provide feedback.

The second step is to examine the experience beyond just summarizing what happened and trying to look at the relationship between civics and learning. This step is designed to help diverse learners understand issues of privilege and power, compare the individual and public good, and explore the dynamics of agency. Given the complexity of this examination, instructors may consider allowing students to write this reflection in the L1. Assessing this step would then need to focus on the students' attention to detail, insights, quality of expression, etc. Instructors may consider having students record a presentation based off this step and share it on their learning management system for other students to view and comment.

Finally, the last step is the articulation of learning in which the learners develop goals for "future action that can then be taken forward into the next experience for improved practice and further refinement of learning" (2009: 42). These goals can be written down and then shared with the class with the students explaining their choice regarding the different goals. Clifford & Reisinger (2019) state that this final step allows the learner to answer four important questions: "(1) What did I learn? (2) How did I learn it? (3) Why does it matter? and (4) What will I do in light of it?" (2019: 71). CBLL provides the ideal environment to allow students to not only use their language skills but see how it directly impacts the community and specific individuals.

Summarizing the benefits of CBL, Clifford & Reisinger (2019) declare that CBL "provides opportunities to expand interpersonal, intrapersonal, and cognitive domains in student development. Students learn more tolerance for ambiguity, dismantle stereotypes, build compassion, and establish reciprocal and authentic relationships" (2019: 28–29). This is also inclusive of diverse communities who benefit from these interactions and reflections not only about their own language skills but also about the community and culture that surround them.

Gregory L. Thompson

### **3 Integrated Performance Assessments (IPAs)**

According to Wiggins (1998), "the aim of assessment is primarily to educate and improve student performance, not merely to audit it" (1998: 7). Additionally, student assessment should be structured around authentic, real-life activities that are interactive and engaging for learners. One movement to try to achieve these goals has been through the greater use of Integrated Performance Assessments (IPAs). IPAs are defined as "ongoing, formative, and standards-based assessments that connect what is taught to what is learned and assessed and that provide the student with detailed and appropriate feedback" (Adair-Hauck & Troyan 2013). Diaz Maggioli (2020) further describes IPAs as "a form of cluster assessment which capitalizes on the inherently intertwined nature of the three modes of communication: interpretive, interpersonal, and presentational" (2020: 24).

The history of IPAs goes back to a project carried out by American Council on the Teaching of Foreign Languages (ACTFL) that used federal funding to design the IPA prototype in response to a high demand for standards-based assessments. The IPA prototype was designed to measure students' progress towards reaching the ACTFL World-Readiness Standards. IPAs were created to assist instructors in connecting standards-based classroom instruction and assessment practices, so the two continuously coincided in the language classroom. The IPA prototype was to serve as a catalyst for curricular and pedagogical reform. ACTFL wanted to show educators how to properly connect assessment with practice so that they were not seen as separate identities in language learning.

Early research into performance assessment pre-date the development of IPAs and the integration of the ACTFL World-Readiness Standards. Pierce & O'Malley (1992) looked at the value of using performance assessments with language minority students. They describe how using a variety of performance assessments with diverse learners not only helped them to increase their participation but also improved the assessment of their learning. They conclude stating, "To be able to effectively monitor the progress of language minority students, assessment needs to be conducted on an ongoing basis with procedures that promise to yield the most useful information for classroom instruction" (1992: 27).

Diaz Maggioli (2020) describes current research into using IPAs and how to help students increase their performance. He found that students often struggle with interpretive and interactive tasks due to their lack of exposure to authentic listening sources and opportunities to interact with native speakers. Frequently students are only receiving input from their instructors and from modified audio sources. In addition, most of their conversation are with fellow second language

### 2 Current trends in language assessment

learners who often struggle with the same issues that they have and are not able to help them make the necessary corrections to their speech.

In addition to IPAs helping students learn, Adair-Hauck et al. (2006) conducted a study of over 1000 students as well as 30 foreign language instructors to determine the impact of IPAs on the instructor's perception of learning. The participating instructors reported that the IPAs:

served as a catalyst to make them more aware of the need to integrate the three modes of communication into their lessons on a regular basis, design standards-based interpretive tasks using authentic documents, integrate more interpersonal speaking tasks, use more open-ended speaking tasks, and use more standards-based rubrics to help the students improve their language performance. (2006: 373)

Thus, the implementation of the IPAs helped to make the instructors not only more aware of what they needed to be doing in the classroom with their students but also helped to focus them on a more standards-based approach to language learning. The IPAs are able to move students and instructors from viewing language as the acquisition of a single skill to an interconnected approach of integrating the different language modalities together to acquire a language.

Troyan (2016) states that in selecting the appropriate listening segments and reading passages that educators should consider two important factors "(1) learner-based factors (e.g., linguistic level and age) and (2) text-based factors (e.g., context and the task related to the text)" (Troyan 2016: 171). Considering these factors can help guide instructors to be more decisive in the materials that they use in their classes and improve their students' experiences with them. Adair-Hauck & Troyan (2013) mention the following resources as examples of where to find authentic sources appropriate for specific learners:


### Gregory L. Thompson


IPAs can also include an element of CBLL where students can interact with native speakers and reflect on these interactions. Since IPAs focus on interpersonal, interpretive, and presentational speaking, using CBLL highlights the interactive, sociocultural nature of language learning and moves it beyond just listening and understanding (input).

The benefit of this type of assessment for learners is the exposure not only to authentic sources and speakers, but an opportunity to engage with diverse communities and develop intercultural awareness and sensitivity. One can imagine a classroom where the students are participating in CBLL and thus receiving authentic input and engaging with native speakers in the second language. They are then coming back into the classroom where they are presenting and reflecting on their interactions as well as considering some of the struggles, which they have had both with the language and with any cultural misunderstandings through IPAs. These students can then work with their instructor to practice the areas where they need to improve and develop a plan based off the three modes of communication that they are using. This type of constant and constructive feedback and performative assessment would assist the students in understanding their own language development and growth.

Adair-Hauck & Troyan (2013) summarize their research on IPAs declaring, "The IPA provides useful information to both the teacher and the learners regarding the kinds of authentic tasks the learners can perform across the three modes of communication and what the learners need to do to improve their language performance" (2013: 37). In order to assess students' growth and development with the IPAs, ACTFL's performance descriptors would be excellent criteria that could be used with different levels of proficiency depending again on the class level. These performance descriptors could be used with the different phases of the IPA to help students with more formative assessments. ACTFL also has the can-do statements that could serve as a baseline for measuring what students are able to do and could be applied to some parts of the IPA. Finally, the Common European Framework of Reference for Languages (CEFR) also has common reference levels that would be good criteria for looking at student growth especially in regards to language proficiency.

### 2 Current trends in language assessment

### **4 Portfolios**

Portfolios have existed and been widely utilized in many professions such as art, architecture, photography, journalism, etc. (Lam 2017). The use of portfolios in the language classroom also has a rich tradition that has continued to evolve as technology changes the ways in which they are developed and presented (Fox 2016, Lam 2017, McMillan 2018). McMillan (2018) defines portfolios as a "purposeful, systematic process of collecting and evaluating student formative and/or summative assessments to document progress toward the attainment of learning targets or show evidence that learning targets have been achieved" (2018: 303). One of the keys of portfolio assessments is the ability to demonstrate progress (formative assessment) of students over a period of time even though they can be used for summative assessment as well. Hamp-Lyons & Condon (2000) declare that portfolios involve three phases: collection, selection, and reflection.

There are several benefits to using portfolios as a tool for assessment. They help create a match between classroom activities and assessment. Students will be able to better understand what is going on in the classroom and then be assessed in a way the reflects their learning. They also capture a rich array of what students know and can do without focusing too much on what students cannot do as in many traditional assessments. Along these same lines, portfolios chronicle students' language development over time and show their progress. This allows students to highlight where they started from and where they have reached (Genesee & Upshur 1996). This allows for more differentiated assessment since summative assessments often do not recognize growth especially in struggling students.

Portfolios also allow students to evaluate their own work, effort, strategies, goals, and progress as these assessments require self-assessment and reflection. Students are able to explain their growth and take responsibility for their own learning. They are also able to better understand how grades are represented as they compile and consider their own portfolios. Since portfolios are often formative, they allow students to establish ongoing goals and review their progress towards the goals they have established (Tedick & Klee 1998).

In portfolios, students are able to demonstrate their overall proficiency both in regards to language and culture and portfolios can even empower students to become their own advocates for their learning as well as for their assessment (Alam & Akar 2019). Portfolios allow students to explain their learning in a way that is collaborative in nature with their instructor leading to greater language acquisition. Finally, (Tedick & Klee 1998) explain that portfolios are not limited to one language modality but represent "a student's range of performance in reading,

### Gregory L. Thompson

writing, speaking, and listening as well as cultural understanding" (1998: 20). The assessment of portfolios can involve a variety of individuals including peer assessment, self-assessment, collaborative assessment, and instructor assessment. Tedick & Klee (1998) observe:

Determining how to go about assessing portfolios in a systematic way is a process that involves reflection, much discussion and negotiation with students and colleagues, and risk-taking. The more the collaboration, the better the process, and, most certainly, the outcome. (1998: 22)

Lam (2017) states that the rationale for implementing portfolio assessment is that assessment "should be personalized, longitudinal and contextualized, taking place in learners' familiar classroom environments rather than being dehumanized and standardized, administered in the examination hall" (2017: 85). This personalization of learning is valuable to language learners and makes them feel part of the process of language acquisition. Tedick & Klee (1998) declare, "The evaluative process should include ongoing (formative) assessments of students' work as well as overall (summative) assessments" (1998: 21).

McMillan (2018) describes four types of portfolios that instructors can use to assess their students. He classifies three of the types as documentation portfolios (celebration/showcase, competence or standards-based, and project) with the other category being growth portfolios. All of these portfolios can be done in the TL and often are since they are related to the work in the course. The reflections can be completed in the L1 or TL depending on the level of the students and the goals of the course.

Celebration/showcase portfolios are compiled to show a student's work that illustrates achievement and highlight some exceptional part of learning. In these cases, the student often selects their best work or what they are most proud of to share with the instructor and/or class. Since each student picks what information they want to highlight, each individual portfolio is unique and personalized to the individual. While this allows for a great deal of creativity and individuality, it also complicates the scoring of each portfolio and can make reliable scoring a challenge especially across a large classroom.

The second type of documentation portfolio that can be used to assess language students is a competence or standards-based portfolio. McMillan (2018) defines this type of portfolio as being designed "to provide evidence that a targeted level of proficiency has been achieved. For this kind of portfolio, the criteria for determination of mastery or competence need to be clearly defined" (2018: 304). The competence or standards-based portfolio is one that is designed

### 2 Current trends in language assessment

to collect a wide range of evidence regarding the proficiency level of the students for a specific class or program. Evidence can be collected based off all language modalities and representative of the overall competence of a language learner.

The third type of documentation portfolio is the project portfolio. McMillan states that the main objective of these types of portfolios is to provide a "single example or illustration of the competence of the student" (2018: 304). Students compile these portfolios with a very specific task in mind and work towards assembling these with the mindset of highlighting some specific aspect of their learning. The final type of portfolio mentioned by McMillan is the growth portfolio. The growth portfolio is a formative assessment that can be used to assess the changes in the proficiency level of students over time. These types of portfolios are beneficial in documenting changes in students' language skills and also provide examples to allow students to see their own growth in the skill sets that they possess.

Wewer (2020) studied how language portfolios were being used in the European context where the implementation of the European Language Portfolio (ELP) was developed in 2001 in cooperation with the Common European Framework of Reference (CEFR) for language learning. According the ELP website (https://www.coe.int/en/web/portfolio), the European Language Portfolio (ELP) was developed by the Language Policy Programme of the Council of Europe (Council of Europe 2001) to support the development of learner autonomy, plurilingualism and intercultural awareness and competence; to allow users to record their language learning achievements and their experience of learning and using languages.

In spite of the implementation of the ELP, Wewer (2020) found that out of all of the different types of assessments both traditional and alternative that "the least used assessment method by teachers was the language portfolio" (2020: 150) even though it has been around since 2001. She also found that other alternative assessments such as simulations, peer assessments, and graded presentations were also among the least common assessments with most instructors opting for traditional assessments or standardized assessments. Wewer (2020) further states that the lack of use of alternative assessments "calls the serious question of whether or not teacher-based, formative assessment is genuinely used for the purpose of enhancing learning" (2020: 150). She discovered that instructors' intentions for language assessment differed dramatically from their actual assessment practices. She declares:

The cornerstone of any approach to assessment promoting learning in CLIL is to make the learners aware of the dual learning objectives (content and

### Gregory L. Thompson

language), their own learning processes, what is already learnt, and how they themselves can further promote and advance the attainment. Such an action necessitates communication and feedback. One means to this end could be the least used assessment method reported by teachers, the language portfolio. (2020: 160)

In spite of its lack of generalized use in the European context, Wewer (2020) notes that in classrooms using portfolio assessments both parents and students found them to be useful and enjoyable as well as being good representations of students' language skills. She also comments that those classes that employed language portfolios as part of their assessments had slight increases in their proficiency in the TL.

Regarding the assessment of portfolios, Tedick & Klee (1998) recommend that instructors not only have their students compile the portfolio but also should ask students to reflect on question related to their overall understanding of their learning process and struggles. Students may be asked to reflect on their acquisition of language and culture as well as how they have contributed to their learning. The assessment of the portfolio would be based on a rubric to analyze both the quality of the portfolio and the reflection. Kunschak (2020) suggests combining portfolio assessments with other testing measures. She recommends measures that could include:

standardized test scores but also evidence of achievement of learning outcomes such as papers or videos or other authentic samples of performance tied to a specific rubric of learning objectives (e.g., a term paper on a policy issue or a group presentation on a mini-research project). Evidence of progress such as multiple drafts, peer review sheets, and reflective comments or diagnostic, mid-term, and final in-class timed writings could also be included. (2020: 99)

She goes on to say that "evaluations of innovative programs need to be cyclical like action research, moving from planning to implementation, assessment to reflection and on to the next round" (2020: 99).

Delett et al. (2001) provide several steps necessary for successful portfolio assessment. They state that the first step is to plan the assessment purpose. This coincides with the different types of portfolios mentioned earlier in this chapter where an instructor needs to determine the reason for choosing a portfolio assessment. The second step is to define the portfolio outcomes as these are important to help the learners and instructor focus on the skills and knowledge that

### 2 Current trends in language assessment

they hope students will acquire. The third step is to match the classroom activities with the established outcomes. Since the portfolio assessments often consist of assignments from the class, it is important to make sure that the classroom activities generate the necessary materials for the portfolios and that they are articulated to maximize learning. The fourth step is to establish the organization of the portfolio. Having confusing instructions or not being clear regarding the content of the portfolio can make this a negative experience for the students as they will not be sure what to include in their portfolios. Fifth, the instructor needs to clearly establish grading criteria so that students know what is expected and what represents excellence in their portfolios. Establishing clear rubrics for every aspect of the portfolio assessment will make the overall grading both transparent and fair. Sixth, the instructor needs to make sure to monitor students' progress throughout the whole process. If an instructor waits until the end of the unit or course to finally see the portfolios, it is likely that some students will have misunderstood and at that point, it will be too late to make meaningful changes. Finally, the instructor needs to monitor and reflect upon the whole portfolio process not only to make changes during the semester but also from semester-tosemester and year-to-year. As certain assignments work and others do not then the instructor needs to be cognizant of the needed changes and make them. All of these steps can lead to making portfolio assessment a valuable tool to understand students' growth and learning in the language classroom. Portfolio assessment can also help the instructor to see how daily activities in class need to reflect the overall language learning objectives.

### **5 Conclusion**

Briefly outlined in this chapter are three alternative assessments which can greatly serve diverse learners with more contextualized settings for learning and assessment (see Appendix for further resources). Using CBLL, IPAs, and portfolios can move assessment from simply recalling and repeating information gleaned from classes and readings to the real application of language skills and abilities with authentic communities and through authentic resources. As Kunschak (2020) states:

By integrating content and language in assessment, students can be subtly guided towards a more holistic approach to learning or deep learning with a view to applying their skills rather than studying vocabulary for a test or memorising concepts by heart. (2020: 98)

### Gregory L. Thompson

While the scope of this chapter only allows for a sampling of the many ways these assessments can be used both in and out of the classroom to promote student learning and development, these should help instructors by providing them with some ideas on where to begin to implement changes in the way students are assessed. Additionally, instructors can also consider the many ways in which all three of these alternative assessments can be complimentary to each other and could be used together as both formative and summative assessments of students' language and culture development. Keeping the students learning outcomes in mind, instructors can revisit their current forms of assessment and determine where they may be able to make changes to better help their students become more competent language learners.

### **Appendix: Additional resources for alternative assessment development**

This following appendix contains additional resources with more information, rubrics, additional examples, and other help needed for developing these tools to be used in the classroom.

### **Community based language learning**


### 2 Current trends in language assessment


### **Integrated Performance Assessments**


### Gregory L. Thompson

• Adair-Hauck, B., and Troyan, F. J. (2013). A descriptive and co-constructive approach to integrated performance assessment feedback. *Foreign Language Annals*, *46*(1), 23–44. **–** This article has several appendices with materials useful for developing IPAs.

### **Portfolio assessments**


### **References**


2 Current trends in language assessment


### Gregory L. Thompson


2 Current trends in language assessment


## **Chapter 3**

## **Informing curricular alignment through alternative assessments: The role of learner self-perceptions in Spanish second and heritage language programs**

### Eva Rodríguez-González<sup>a</sup> , Marián Giráldez-Elizo<sup>b</sup> & Sarah Schulman<sup>c</sup>

<sup>a</sup>University of New Mexico <sup>b</sup> Santa Fe Prep High School <sup>c</sup>New Mexico International School

The present quantitative cross-sectional study examines how Spanish language learners in Beginning and Intermediate courses at a university in the U.S. Southwest judge their abilities to organize and perform interpersonal and presentational speaking and writing tasks in the target language. The data set includes self-assessment survey responses from a total of 133 Spanish language learners enrolled in first- and second-year General Education courses. These individuals are matriculated into two different language programs based on their academic or home and/or community exposure to the Spanish language. Participants therefore include Spanish as Second Language learners (SSL, N = 67) and Spanish as Heritage Language Learners (SHL, N = 66). Participants ranged in proficiency from Novice High to Advanced Low and responded to a Can-Do Statement questionnaire (NCSSFL 2014) that was directly aligned to course objectives. Results for the interpersonal speaking and presentational writing domains suggest that participants statistically differed in their self-efficacy based on their language program. In response to this self-efficacy variability, the chapter includes a sample lesson plan that situates inclusivity, equity and diversity as the foundation for all classroom activities. The *Can-Do Statements* are also incorporated to illustrate how conscious awareness of one's language abilities can foster learner autonomy and inform programmatic assessment.

Eva Rodríguez-González, Marián Giráldez-Elizo & Sarah Schulman. 2022. Informing curricular alignment through alternative assessments: The role of learner selfperceptions in Spanish second and heritage language programs. In Eva Rodríguez-González & Rosita L. Rivera (eds.), *Language assessment in multilingual settings: Innovative practices across formal and informal environments*, 35–65. Berlin: Language Science Press. DOI: 10.5281/zenodo.6762274

### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman

### **1 Introduction**

### **1.1 Self-assessment via self-efficacy of language learning**

Student self-assessment in language classrooms occurs when learners assess their own performance, and it is primarily used to help students develop specific learning skills they will need for communicative and intercultural competence. This process may assist in making learners more aware and responsible for their own learning process. Learners' self-assessment practices embedded in Higher Education learning via measurement of self-efficacy have become increasingly popular since the early 2000s (Papanthymou & Darra 2018). Documenting language development helps learners to: 1) *develop* important metacognitive skills that will allow learners to evaluate their own performance 2) *increase* self-awareness through reflective practice 3) *reinforce* the development of critical reviewing skills through peer evaluation, and 4) contribute to learners' autonomy. That being said, self-assessment via self-efficacy gives learners a greater amount of agency regarding assessment, thus enriching their learning. Skilled self-assessment can be as reliable as other forms of assessment; however, instructors must provide learners with guidance and practice so that the results of these tools closely align with the results from other assessment agencies (e.g. instructors and/or program/degree evaluation).

In terms of application and measurability of self-efficacy in language classrooms that emphasize meaningful, communicative learning tasks, self-assessment has been recommended as a way to have learners reflect upon their own learning and make judgments of their own performance in the target language (Klein 2007). *LinguaFolio* is a self-monitoring learner portfolio tool that enables goal setting and collection of evidence of language achievement. It was specifically created for measurement of progress and growth in second languages other than English in the context of the U.S. As such, *LinguaFolio* can serve as a type of assessment as it contains a set of multiple language learning standards that have been adapted into classroom goals as "can do" statements that follow the *American Council of Teaching Foreign Languages* (ACTFL) proficiency guidelines. *Can-Do Statements* have been shown to increase learner motivation, language proficiency, and academic achievement (Collett & Sullivan 2010; Moeller et al. 2012). Although *Can-Do Statements* were originally designed to enhance the learning of second language learners, the same guiding principle can also be applied to heritage learners (Cox et al. 2018: 106). The question for educators is how to draw on this information to better respond to the needs of their learners. By identifying learners' perceived learning abilities, language teachers can better target their instruction to support learners' developing linguistic proficiency (Hlas 2018: 49).

### 3 Informing curricular alignment through alternative assessments

Since the integration of *Can-Do Statements* in second language learner (L2) language classrooms promotes a reflective learning process that is directly correlated with goal setting and self-assessment, language teachers should consider how to provide guidance to learners toward self-regulated and autonomous learning (see Moeller & Yu 2015 for a detailed description of *Can-Do Statements*). Given that speaking and writing are often identified by language learners as the most difficult skills to learn, the present study focuses on learners' self-assessment of their capabilities in these two modalities (Aida 1994; Cheng et al. 1999; Phillips 1992).

### **1.2 Self-efficacy: Bandura's theoretical framework**

In the context of a language classroom, feeling prepared to communicate verbally in a given situation and being able to engage in a conversation to successfully navigate that situation illustrate the difference between a speaker's outcomeexpectancies and efficacy. The psychological motivation an individual requires to overcome their performance doubts is what Bandura (1977) referred to as *selfefficacy*. To elaborate, self-efficacy concerns "people's beliefs about their capabilities to produce designated levels of performance that exercise influence over events that affect their lives" (Bandura 1994: 2). When an individual believes that they no longer have control over the outcome of a particular event, they may begin to doubt their capabilities. This negative thinking can then culminate in feelings of self-sabotage, lower aspirations, and depression (Weibell 2011). To analyze changes in fearful and avoidant behavior, Bandura (1997) situates self-efficacy as central to his theoretical framework.

As a cognitive process, self-efficacy will vary according to an individual's lived experiences. Bandura (1997) contends that certain types of self-efficacy can positively influence how individuals approach a given task. These include:


Experiences that are self-fulfilling and lead to a sense of accomplishment are most effective at enhancing self-efficacy. Since the application of Bandura's (1986)

### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman

view of human behavior has been documented for educational contexts, and most particularly, its application to language learning environments (Wenden 1998), Bandura's theoretical framework plays a pivotal role in the language classroom, as learners are constantly engaging in performance tasks that test the limits of their perceived self-efficacies. As such, self-efficacy in language classrooms is directly related to a learner's belief or self-assessment about his or her own competence to perform specific tasks (Bandura 1986, 1997). Thus, each learner's sense of self-efficacy can play a major role in how s/he approaches goals, tasks, and challenges. More importantly, whether learners perceive themselves as capable of doing a given task can predict their performance outcome more often than their real abilities (Bandura 1997).

### **1.3 Self-efficacy in language learning classroom**

Significant research regarding self-efficacy and other variables in second language learning classrooms, such as learning strategies, performance, and language anxiety, has emerged only within the last few years (Raoofi et al. 2012: 61). Still missing from these studies is an exploration on how learners' reported self-efficacies for each specific language skill (i.e. listening, reading, speaking and writing) relate to the modes of communication (i.e. interpretive, presentational and interpersonal) across different learner proficiency levels (Torres & Turner 2016). For example, in classrooms where tasks "involve communicative language use in which the user's attention is focused on meaning rather than grammatical form," (Nunan 2004: 4) learners may perceive some activities in the target language as more challenging than others (e.g., gap filling activity vs. creating a short story in Spanish). Consequently, learners may differ in their reported self-efficacies for completing those tasks (Torres & Turner 2016).

### **1.4 Self-efficacy in second language speaking**

One of the most important variables language educators must consider is how learners' self-perceived capabilities can impact their performance on a given language task. Furthermore, the nature of the task, whether it is reading, writing, listening, or speaking, will influence a learner's degree of reported self-efficacy (Dewaele et al. 2008; Horwitz 2001; Kim 2009; MacIntyre et al. 1997; Phillips 1992; Raoofi et al. 2012). This aligns with other research regarding how more private tasks, such as reading, elicit lower levels of anxiety, and therefore, result in higher levels of self-efficacy. As such, learners are more objective at evaluating themselves in reading than any of the other language skill, as the privacy allows for learners to escape possible judgement by others (MacIntyre et al. 1997: 279).

### 3 Informing curricular alignment through alternative assessments

Conversely, speaking tasks have a profound impact on learner self-efficacy, as the absence of privacy tends to produce higher levels of anxiety (Ellis 1994; Horwitz et al. 1986; Young 1991; as cited by Cheng et al. 1999: 418). The fear of making grammar mistakes or mispronunciations in front of others is consistent across all levels of linguistic proficiency (Aida 1994; MacIntyre et al. 1997; Phillips 1992). Even learners who are raised in a home where a non-English minority language is spoken may be apprehensive about speaking the target language in front of their peers. They may also feel that their actual performance may not align with classroom expectations.

To illustrate, Kim (2009) studied the anxiety levels of Korean learners of English enrolled in conversational and reading courses and determined that learners enrolled in the more communicative courses experienced higher levels of anxiety because of a "fear of negative evaluation" when speaking spontaneously or in front of others (2009: 153). When anxiety levels increase, learners are more likely to depreciate their self-efficacy on various learning tasks (MacIntyre et al. 1997). However, while a number of empirical studies have focused on the relationship between language learning anxiety in speaking and learner self-efficacy (Horwitz 2001; Horwitz et al. 1986; Phillips 1992; Woodrow 2006), it is important to note that interpersonal communication tasks also include writing (Cheng et al. 1999).

### **1.5 Self-efficacy in foreign language writing**

While research on writing anxiety is often limited to the study of English as a first language in the U.S., Cheng et al. (1999) argue that fear of evaluation can still influence learner performance on a specific writing task. Specifically, the writer's perceived quality of his encoded message can cause him to doubt his academic writing skills, which may, inadvertently, hinder his career choices (Daly & Miller 1975; Daly & Shamo 1976). Self-efficacy in speaking and writing tasks are therefore not to be collapsed into similar categories. Rather, writing apprehension is unique to the written domain. As Woodrow (2011) notes, however, "there is relatively little research on the relationship between self-efficacy and writing that elucidates how learners' perceptions of their abilities influence their performance on written language tasks" (2011: 511).

To better ascertain this relationship, Woodrow (2011) conducted a study that examined the interplay of learner self-efficacy and anxiety levels when writing in English as a second language. Her results suggested that self-efficacy had a more significant impact on predicting students' language learning and performance than learners' feelings of anxiety. However, it is important to note that

### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman

learners who were more anxious had lower self-efficacies, and consequently, they tended to focus more on the importance of an assessment than the value of their own learning. This fear of failing to meet expectations in oral or written production of a target language was particularly evidenced among learners who were home and community speakers of this language. These heritage language learners added further complexity to understanding and responding to learners' reported levels of self-efficacy, as their linguistic capabilities varied tremendously.

### **1.6 Self-efficacy in heritage language learners**

When referencing home and community speakers of the target language, the term *heritage language learner* (HLL) is often used. However, it is important to note that HLL is a widely recognized yet often misunderstood concept. To this day, there is an absence of a definition that fully captures the term's historical, sociocultural, and psychological complexity. Valdés (2001) provides the most frequently referenced description, explaining that a heritage language learner is "a language student who is raised in a home where a non-English language is spoken, who speaks or at least understands the language, and who is to some degree bilingual in that language and in English" (2001: 38). HLLs therefore have diverse range of communicative and cultural experiences, which manifest in linguistic and affective needs that differ from those of traditional second language learners. Examples of the differences between these two learner groups include academic achievement and motivation for enrolling in language courses; these differences are frequently noted in the heritage language literature (Hedgcock & Lefkowitz 2016; Torres & Turner 2015; Tallon 2009).

Given the variability of how and when HLLs are exposed to and have acquired some of their language, HLLs often exhibit learning gaps that are not evidenced in second language learners, such as a comprehensive understanding of academic register and metalinguistic knowledge (Hedgcock & Lefkowitz 2016; Torres & Turner 2015). Nonetheless, given that HLLs are typically exposed to the target language at a young age, language educators often assume they will perform equally to or better than L2s on communicative tasks. When HLLs are unable to meet academic register or demonstrate metalinguistic knowledge, this experience can exacerbate deep-seated feelings of inadequacy and frustration (Coryell & Clark 2009; Tallon 2009). The pervasive issue, as Hedgcock & Lefkowitz (2016: 2) explain, is that "traditional approaches to foreign language (FL) instruction rarely target the unique educational needs of heritage language (HL) speakers who represent diverse linguistic, cultural, educational, and socioeconomic profiles." This pedagogical mismatch, in conjunction with the aforementioned factors, is likely

### 3 Informing curricular alignment through alternative assessments

leading to lower ratings of self-efficacy on specific language learning tasks that are also experienced by L2 learners.

Empirical studies that focus on the needs and perceptions of HLLs (Hedgcock & Lefkowitz 2016; Torres & Turner 2015; Tallon 2009) have found through the use of the *Foreign Language Classroom Anxiety Scale* (FLCAS) that HLLs typically experience greater anxiety with tasks that require reading and writing and less anxiety with those that require speaking. L2s, by contrast, exhibit greater anxiety with more public tasks, such as interpersonal and presentational speaking (Mac-Intyre et al. 1997). As previously mentioned, anxiety is an important variable that challenges learners' self-efficacies. In one study, Tallon (2009) found that self-efficacy ratings among HLLs fluctuated according to the number of complex grammatical structures. That is, the greater the number of language constructs under study, the greater the learners' level of anxiety. These findings suggest that instruction that helps HLLs capitalize on their current skill set is paramount. It is therefore critical that instructors learn about the language experiences and cultural connections of their heritage students in order to create HLL-specific learning goals and tasks that validate and promote the learners' home varieties. These goals may include learning specific vocabularies, as well as academic written and spoken registers, that are related to places of employment. By incorporating students' backgrounds and learning interests into the curriculum, instructors can create a safe space in which HLLs can "share and discuss their language experiences" (2009: 125). Such intentional planning with regards to meaningful learning objectives and activities will help build confidence and increase learner self-efficacies by fostering a sense of community within the classroom. However, more research is needed to understand how HLLs' self-perceptions of their linguistic and cultural backgrounds influence the way in which they 1) respond to instruction and produce grammatical structures and 2) develop an awareness of and response to their self-reported abilities.

The following study thus explores how L2 and HLLs in two Spanish programs at an institution of higher education in the U.S. Southwest evaluate their capabilities on a variety of *Can-Do Statements* in the oral and written domains.

### **1.7 Can-do statements in language instruction and learning**

*Can-Do Statements* are Performance Descriptors that have been modified into classroom objectives where students can decide whether they can or cannot use the target language in the interpretive, presentational and/or interpersonal mode of communication. *Can-Do Statements* were designed as self-assessment checklists created by the *National Council of State Supervisors for Languages* (NCSSFL)

### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman

that uses ACTFL's Proficiency Guidelines and levels for language proficiency. The NCSSFL-ACTFL 2015's *Can-Do Statements* (Moeller & Yu 2015) were created for students to measure their intercultural communication. For instructors in language programs, these *Can-Do Statements* serve as performance indicators by identifying learning targets for curriculum and unit design. For the language learners per se, these *Can-Do Statements* checklists provide a chart to monitor their progress through incremental steps within a given period of time. In the classroom setting, instructors can implement or adapt these goals in the form of a short survey for students to complete before and after a lesson plan or a relevant language objective. The implementation of *Can-Do Statements* is very valuable in the Spanish language classes as they offer an action-oriented approach that facilitates the integration of task-based pedagogical interventions. Task-Based teaching encourages student-centered activities that use real language in real-world tasks (Long 1985; Norris 2009). The inclusion of *Can-Do Statements* together with a Task-Based pedagogical intervention in the Spanish classroom not only facilitates learners' self-efficacies but also enhances their Spanish learning because learners build their language skills by putting into practice meaningful and real tasks. Additionally, *Can-Do Statements* can be aligned with lesson plans to work on those language objectives that need more development – as well as to incorporate elements of global and community relevance helping to embrace diversity, inclusion, and equity as part of the curriculum.

### **1.8 Self-efficacy in language classrooms: fostering diversity, equity and inclusion**

With the increase in diversity at institutions of higher education, campus communities are often comprised of learners with a wide array of backgrounds and diverse experiences, as well as multiple and intersecting identities and language profiles. In addition, minority members of campus communities have historically been underrepresented. It is in this context where we must foster inclusion and equity by creating an active learning environment which considers the importance of diversity and community well-being and that also constructs understanding on how individuals connect and interact with each other, the systems and institutions. Issues related to identity, background, and linguistic differences manifest across all courses, assignments, curriculums, and pedagogies. Given that instructors play a major role in creating a learning environment that promotes diversity, this section proposes a lesson plan for addressing issues of diversity and inclusivity in mixed language classrooms where different degrees of self-efficacy interplay among language learners. More specifically, the proposed

lesson plan includes a topic pertaining to immigration issues that is socially relevant to Latinx communities in the U.S. and in Latin America. The lesson plan therefore serves as a way to involve learners in the participation and contribution to "a common space for learning" that is based on their own and related communities.

### **2 The present study**

The present study contributes to existing research on the exploration of selfefficacy within the context of language learning in university settings. Specifically, the authors investigate Spanish language learners' self-efficacy of their linguistic proficiency in interpersonal speaking and presentational writing via the use of the ACTFL Can-Do Statement checklist questionnaire. The results subsequently extend previous findings that highlight the importance of self-assessment in determining course goals and language performance in specific communicative domains within a spectrum of different levels of coursework.

Participants in this study were enrolled at a U.S. Southwest university in four different levels of coursework within two different Spanish language programs: Spanish as a Second Language and Spanish as a Heritage Language. As such, all are Beginning (first semester) and/or Intermediate L2 and HL Spanish learners. The study sought to determine to what extent learners perceive themselves as more proficient in Spanish speaking or writing as they progress through different periods of coursework and to see whether differences in self-efficacy exist between L2 or HL. To that end, the following research questions were explored:


### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman

In terms of any potential differences in self-efficacy between Spanish L2 and HLLs, the authors predict lower levels of self-efficacy in speaking tasks as compared with writing for Spanish L2 learners due to high levels of anxiety reported for L2 speaking language tasks (Mills et al. 2007). On the contrary, higher selfefficacy in speaking tasks is predicted for Spanish HLLs due to community language exposure (Valdés 2000).

Following a presentation of the findings, the study provides a descriptivecorrelational report of self-efficacies as portrayed by different language learners (Spanish L2 and HLLs) enrolled in two Spanish language programs (Spanish as a Second Language and Spanish as a Heritage Language programs). The chapter concludes with a proposal of diverse, equitable and inclusive activities that would benefit both kinds of language learners.

### **3 Methodology**

### **3.1 Participants**

All participants (N = 133) were enrolled in Beginning and Intermediate Spanish courses. The selected first- and second-year general education Spanish program consists of two parallel programs and coursework that targets both *Spanish as a Heritage Language Learners* and/or speakers (henceforth SHL) and *Spanish as a Second Language Learners* (SSL). The SHL program is designed for learners who come from communities in which Spanish is traditionally spoken, even if only by the older generation. This program focuses on the revitalization and maintenance of the Spanish language and draws upon the learners' personal connection to the language and culture in order to increase learner motivation. The SSL program is designed for learners of Spanish who are not from Spanish speaking communities or homes, and as such, are learning Spanish as a second language. The distribution of participants recruited from the different programs and coursework is identified in Table 1 below.

Sixty-one percent of the participants were female, while 39% were male. Participants' age predominantly ranged between 18–20 years old (71%, followed by 21% within 21–25 years old and 8% 26–older). In terms of ethnicity, 63% identified themselves as Hispanic, 30% identified as non-Hispanic, and 7% did not specify their ethnicity. In addition to these preliminary background questions conducted via written questionnaire, we also asked participants about their experience and exposure to languages since they were born. Most participants (79%) identified English as their first language, 10% identified Spanish as their first language, and 11% identified either French, German or Portuguese as their first language. When

### 3 Informing curricular alignment through alternative assessments


Table 1: Distribution of participants (N = 133) per language program and levels of coursework

asked about their exposure to Spanish, 47% stated having been exposed to Spanish in a school setting, 28% stated a combination of the home and school, and 5% were exposed to Spanish in a study abroad setting.

Given that the focus of the present study concerns interpersonal speaking and presentational writing tasks, we asked participants about individual perceptions of their strongest and weakest language skill(s) both in their first and second/heritage language in order to determine self-perceived competence or preference areas in each language. As for participants for whom Spanish was a second or foreign language, 30% claimed speaking as their weakest skill in Spanish and 33% claimed writing as their weakest. In the case of Spanish heritage language participants, 41% self-reported speaking as their weakest skill in Spanish and 36% identified writing being their weakest.

### **3.2 Materials**

The present study included the following sets of materials: 1) a preliminary set of background questions presented in a written questionnaire (age, exposure and experience to languages, self-perceived strongest and weakest skill in Spanish, etc.); 2) course objectives listed in course syllabi for the semester-long courses in the Spanish Second Language Program and the Spanish Heritage Language Program (four semesters in each program: two Beginning and two Intermediate courses); 3) responses from *Can-Do Statements* written Questionnaire; and 4) excerpts from a selected reading *La Travesía de Enrique* [Enrique's Journey] by journalist Sonia Nazario. The questionnaire was modified from the 2015 NCSSFL-ACTFL language proficiency guidelines (Moeller & Yu 2015). For the present

study we used those *Can-Do Statements* as directly aligned with each course objective that corresponded to ACTFL's language performance descriptors in multiple levels of proficiency (ranging from Novice High to Advanced Low), two output-based skills (speaking and writing) and two modes of communication (interpersonal and presentational). By the time participants were asked to complete the *Can-Do Statements*, they had engaged in multiple interpersonal speaking and interpersonal and presentational writing tasks in and out of the language classroom. The questionnaire distributed to participants did not explicitly state the expected course level of performance (i.e. a linear sequence of statements as corresponding to different levels of performance ranging from Novice Mid to Advanced Low) and was divided into two sections, namely, interpersonal speaking and presentational writing. An example of an interpersonal speaking and presentational writing Can-Do Statement (Novice Level) used in the present study is presented in (1) and (2) as follows:


### **3.3 Procedure**

For each *Can-Do* statement, participants were asked to identify their level of confidence per each Can-Do statement as "I can do it easily and well" or "This is one of my goals." A total of ninety statements were presented in each questionnaire (45 interpersonal speaking statements and 45 presentational writing statements). For each set of statements, five of them corresponded to seven sublevels of proficiency (Novice Low, Novice Mid, Novice High, Intermediate Low, Mid and High, and Advanced Low). In order to identify participants' overall selfreporting of their proficiency levels, we followed an 80% threshold protocol. If participants selected four of the total five statements assigned for Novice-Mid Speaking/Writing *Can-Do* Statements (4/5 of confidence level per sub-level of proficiency) as "I can do it easily and well," they met an 80% threshold and were assigned a "pass" for that specific sub-level of proficiency. To achieve a "pass" for Novice-High proficiency, the participants had to meet an 80% benchmark. If this 80% benchmark was not met, the participant received a "Novice Mid" as their final rating of self-perceived level of proficiency. Thus, participants who met the

### 3 Informing curricular alignment through alternative assessments

80% benchmark for a given sub-level continued to be rated for the next sub-level until the 80% benchmark was no longer met.

### **4 Results**

### **4.1 RQ#1: Student/Learner's perceptions on their abilities of engaging in Spanish interpersonal speaking and presentational writing communication. Summary of perceptions according to level of coursework and language program (SSL vs. SHL)**

### **4.1.1 Interpersonal speaking**

In order to address the first research question of the present study regarding self-efficacy in interpersonal speaking tasks, we compared results in terms of different levels of coursework (Beginning to Intermediate) and different Language programs to determine if statistical differences were obtained. Figure 1 summarizes the findings for learners' perceptions of interpersonal speaking in Spanish in beginning-level coursework.

Figure 1: Distribution of Self-perceived abilities of interpersonal speaking communication in Spanish from SSL and SHL learners in Beginning Spanish coursework (First year: SPAN100)

The horizontal axis includes sub-levels of proficiency and vertical axis relates to the number of learners who self-identified each sub-level. Error bars inform on the variability of data and indicate how precise each measurement is.

An independent samples t-test analysis was conducted to examine mean differences in the self-efficacy questionnaire measure between the first and second

### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman

semester college students enrolled in the Beginning courses of the SSL and the SHL program. Most Spanish learners in Beginning-level coursework reported higher levels of confidence in the Novice to Mid-High sub-levels of speaking proficiency. In terms of differences between programs, learners enrolled in the SSL program predominantly placed themselves within the Novice-Mid continuum (mean = 1.94). SHL learners, however, expressed greater confidence than SSL learners (mean = 3.00) in their interpersonal speaking abilities, placing themselves in the Novice-High range [(63) = −6.19, = 0.00]. Additional independent samples t-test analysis examined mean differences in the self-efficacy questionnaire measure between the first and second semester college students enrolled in the Beginning courses of the SSL program. Second semester SSL Beginning-level learners have higher self-efficacy ratings than their first semester language peers [(32) = −4.76, = 0.00]. However, learners in the first and second semester of the first year SHL Beginning-level courses did not differ in their perceptions of self-efficacy with regard to interpersonal speaking [(29) = 0.56, = 0.58].

In the case of Intermediate-level coursework, the results presented in Figure 2 portray a wider range of self-identified speaking abilities in Spanish. SSL learners mostly identified themselves as feeling capable of engaging in interpersonal speaking tasks at the Intermediate-Low level, with some considerable ratings also reported in Intermediate Mid-High. Learners in the fourth semester of the SSL Program reported higher levels of self-efficacy in the Intermediate-Mid range, as compared to their third semester peers [(31) = −2.81, = 0.01]. SHL learners also identified self-efficacy in interpersonal speaking at the Intermediate-Low range; however, some SHL learners also placed themselves within the Intermediate-High level. A t-test confirmed that there was no statistical difference in self-perceived ability to engage in interpersonal speaking between learners in the third and fourth semesters of Spanish in the SHL program [(33) = −1.35, = 0.19].

### **4.1.2 Presentational writing**

We also compared results in learners' reported self-efficacies in presentational writing across different levels of coursework within the SSL and SHL Programs. Figure 3 summarizes the findings for learners' self-perceptions of their presentational writing capabilities in Beginning-level Spanish coursework.

Overall results indicate that learners in SHL Beginning-level coursework have higher ratings of self-efficacy in presentational writing, as compared to their SSL

### 3 Informing curricular alignment through alternative assessments

Figure 2: Distribution of self-perceived abilities of interpersonal speaking communication in Spanish from SSL and SHL learners in Intermediate Spanish coursework (Second Year: SPAN200)

Figure 3: Distribution of self-perceived abilities of presentational writing communication in Spanish from SSL and SHL learners in Beginning-level Spanish coursework (First year: SPAN100)

### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman

peers [(63) = −2.42, = 0.02]. As shown in Figure 3, the only sub-level of proficiency that did not exhibit overlapping error bars was the Novice-Mid range. The absence of this overlap indicates that more learners in Beginning-level SSL courses exhibited higher levels of reported self-efficacies in presentational writing than their SHL Beginning-level peers.

Independent t-tests between two courses in a given program did not yield significant differences. Learners in the first and second semester of SSL identified a similar sub-level of proficiency in Spanish presentational writing [(32) = 0.77, = 0.45]. In the case of SHL Beginning (first year- semester I and II), learners in the first and second semester did not differ either in their self-perceived abilities in Spanish presentational writing [(29) = −0.89, = 0.38]. In summary, learners in first year courses in both programs perceived themselves as capable of performing tasks with a similar degree of confidence in presentational writing.

In terms of Intermediate-level coursework, there was an overall effect of similarity between SSL and SHL in Intermediate-level coursework with regard to Spanish presentational writing capabilities [(66) = −1.11, = 0.27] with the majority of learners, independently of language program, rating themselves as capable of performing at the Intermediate-Low sub-level of proficiency (Figure 4). When comparing third or fourth semesters of each program, there were no significant differences between self-efficacy in presentational writing among learners in the third or fourth semester in the SSL program [(3) = 1.02, = 0.32]. A similar effect was found among learners in SHL Intermediate (second year) coursework in such a way that ratings of self-efficacy did not differ among learners in third and fourth semesters in the SHL Intermediate program [(33) = −0.82, = 0.42].

A snapshot of variability of ratings among learners is presented in Table 2 with four 100% stacked bar graphs that represent each language program, levels of coursework and modes of communication. There are four semesters in First and Second Year General Education coursework: 1) SPAN101 and SPAN111 = Semester I in Beginning Spanish, First year in SSL and SHL Programs, respectively; 2) SPAN102 and SPAN112 = Semester II in Beginning Spanish First year in SSL and SHL; 3) SPAN201 and SPAN211 = Semester I in Intermediate Spanish Second year in SSL and SHL; and 4) SPAN202 and SPAN212 = Semester II in Intermediate Spanish Second year in SSL and SHL.

The SSL program portrays a clearer path in terms of confidence growth, as indicated by the three colors represented in the figure for interpersonal speaking abilities. Learners in their first semester (blue) initialize rating themselves as Nov-Low/Nov-Mid and continue in their coursework by feeling more capable of engaging in interpersonal speaking activities. Throughout the sequence of

### 3 Informing curricular alignment through alternative assessments

Figure 4: Distribution of self-perceived abilities of presentational writing communication in Spanish from SSL and SHL learners in Intermediate Spanish coursework (Second year: SPAN200)

SSL courses, either a monochromatic bar or a binary colored bar represents the majority of the learners, indicating that variability occurred within two proximate courses (first and second or second and third) and within two sub-levels of proficiency. However, the color sequence in the SHL figure is not as clear as the SSL one in such a way that learners in the first, second, third and fourth semester rated themselves in Intermediate-Low sub-level of proficiency. A similar scenario appears for self-efficacy ratings in the presentational writing mode of communication where a single or two colors are representative of two approximate courses or sub-levels of proficiency in the SSL Program but a wider array of perceptions is plotted throughout the different levels of coursework in the SHL program.

### **4.2 RQ#2: Correlation between learners' perceptions of self-efficacy and course objectives**

Upon data analysis, learners' self-efficacy ratings for interpersonal speaking and presentational writing tasks were compared across respective Spanish courses. The data served as an indirect measure of assessment on how learners' perceived capabilities aligned with specific Student Learning Outcomes (SLOs) identified by instructors at the beginning of each course. The information presented in

Table 2: Vignette of language proficiency trajectory of learners' selfefficacy reported abilities based on different modes of communication and language programs throughout a sequence of four Spanish language coursework

Mode of Communication: Presentational Writing

### 3 Informing curricular alignment through alternative assessments

Table 3: Comparison between self-perceived abilities in speaking and writing and expected ACTFL sub-level of proficiency in course objectives determined by student learning outcomes (SLOs)


the first row of Table 3 above outlines the different expectations of sub-levels of proficiency per course within each program. It is important to note that the SLOs for the SHL program are not necessarily aligned with ACTFL performance descriptors for different levels of proficiency. In order to compare and contrast our findings, we identified ACTFL equivalencies for SLOs in all courses of SHL program. As we see in the second row of Table 3, learners' self-perceptions match with programmatic "expected" SLOs in Beginning coursework of both SSL and SHL programs in speaking and writing. However, the Intermediate coursework expectations and self-perceived abilities of language proficiency in interpersonal speaking and presentational writing do not match. More specifically, the distance between an expected outcome of language proficiency and self-efficacy ratings is considerable in the case of learners in the Intermediate-level Spanish coursework in the SHL program (Advanced Low vs. Intermediate Low). On a holistic view, the results confirm a direct alignment of course objectives and learners' perceptions of self-efficacies in the Beginning-level courses in both programs.

### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman

### **4.3 RQ#3: Inclusion of** *Can-Do Statements* **as self-efficacy tool in daily classroom activities?**

In order to answer the third research question, we propose a sample of a lesson plan that incorporates *Can-Do Statements* in daily language classroom activities in an Intermediate Spanish Language course for SSL and SHL curriculum. Although the proposed instructional lesson plan has been implemented in a Latinx-serving institution in the U.S. Southwest region, it exemplifies how the exploration of current global issues, such as social justice, can be aligned with ACTFL *Can-Do Statements* and applied to broader language learning contexts and classrooms where mixed language learner profiles are represented. Through this lens, the lesson plan instills a sense of awareness regarding cultural and linguistic differences, while promoting social justice in the classroom. The selected reading for the following lesson plan is *La Travesía de Enrique* [Enrique's Journey] by journalist Sonia Nazario, as the content is directly related to immigration issues. The novel addresses the reality of South American people who are leaving their countries behind for a better life in the U.S. As such, the reading is reflective of the numerous stories of Latinx communities coming to the U.S., stories to which many learners can readily recognize and relate. For instance, some learners may have familial immigration histories that are similar to those evidenced in the novel, while others may associate the story's events with immigration issues that they see in the news and other media. The events explored in *La Travesía de Enrique* therefore serve as the framework for the lesson plan. The lesson plan itself facilitates conversations about current political issues that allow students to reflect and critically think on patterns of inequality, discrimination, and injustice. It also empowers learners to examine and question situations that they deem unfair in their lives or the lives around them.

With the focus of the lesson established, the *Can-Do Statements* were then embedded as a pedagogical tool to measure how learners rated their abilities in the different language areas. Important to note is that the *Can-Do Statements* included in the present study were also utilized in SSL courses as a continuous, semester-long self-evaluation assessment. The overall design of the lesson plan thus exemplifies how *Can-Do Statements* can be interwoven to support the exploration of a topic that encompasses global and community issues, thereby establishing inclusivity for both L2 and HL learners.

While data from the *Can-Do Statements* informed the proposed design of lesson plan for learners in Intermediate L2 Spanish program, this theme can also be adapted for HLLs (see Table 5 for details). The *Can-Do Statements* checklists used in the present study were distributed before a lesson plan and were imple-

### 3 Informing curricular alignment through alternative assessments

mented and returned to learners with the appropriate feedback following the parameters of *Linguafolio* and identified ACTFL performance descriptors for the activities and the course.

For this specific sample, we designed the activities based on ACTFL performance descriptors for Advanced Low level of proficiency. Although the ACTFL proficiency level identified in the course (cf. list of student learning outcomes) was Intermediate High, classroom activities have been designed with Advanced Low performance descriptors following Krashen's + 1 comprehensible input principle (Krashen 1982, 1985). As such, comprehensible input is that input which is slightly beyond the current level of competence of a language learner. If *i* is the language learner's current level of competence in the target language, + 1 is the next immediate step along the development continuum. In this particular course, course activities have been designed for Advanced Low level of proficiency as the next immediate sub-level following the identified Intermediate High expected level of proficiency in the course.

Table 4 provides a summary of the student learning outcomes identified for different activities based on speaking and writing language skills. A series of *Can-Do Statements* portrayed in the table correspond to each learning outcomes for speaking and writing. SAL refers to Speaking Advanced Low learning outcomes to be assessed and the number refers to their place in the list. WAL stands for Writing Advanced Low and the number equally refers to which of the *Can-Do Statements* from the list we are referring to. The *Can-Do Statements* have been adapted to activities that 1) reflect on immigration issues in the U.S.; 2) identify key points and reframe them by using learners' own words; and 3) compare primary sources and connect them to lived experiences. In the different activities, learners reflect on immigration, identify key terminology and paraphrase, compare and connect the reading with their personal experiences on the subject matter.

*The lesson plan:* The design consisted of a series of activities for a Spanish intermediate-level course (fourth semester- Intermediate Spanish II). These proposed activities are not time-sensitive and can be used and adapted throughout a given instructional period. The four-day lesson plan described below is not time-bound, as it could be completed in a one to two-week frame or the time the instructor considers appropriate depending on their syllabus and course objectives. Furthermore, each activity is aligned with specific *Can-Do Statements* identified in Table 4 and is connected to the specific mode of communication (i.e. interpersonal, presentational or interpretive) and the particular instructional objective. All the activities in the present lesson plan may be differentiated for L2 and HLLs.

### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman

Table 4: List of *Can-Do Statements* used for self-assessment in the proposed lesson plan (based on ACTFL performance descriptors at Advanced Low level of proficiency)


SAL = Speaking Advanced Low and WAL = Writing Advanced Low. The numbers represent the adapted Can-Do statement numbering from *Linguafolio*.

For this lesson plan, we have identified Advanced Low performance descriptors for the activities with the understanding that instructors must remain cognizant of their learners' linguistic capabilities and align the ACTFL proficiency guidelines with the specific learning needs of their student population (see Table 5). To be consistent with the results presented for RQ 1 and 2, the current lesson plan focuses on self-assessment of speaking and writing as these were the language skills participants in the study reported as feeling less confident in.

The proposed lesson plan in Table 5 includes collaborative learning-based activities that are focused on Latin America-U.S. immigration issues. By engaging in these activities, students learn about aspects of immigration and connect this understanding to the personal experiences of Latin American immigrants. To prepare for the *Final in-class discussion*, instructors may start with a baseline *Prereflection activity* that encourages students to identify what they believe they understand about immigration, as well as highlight potential areas of new information. This preliminary activity encourages students to think critically on why and how individuals and/or families immigrate.

During *Reading in-class activities*, students identify key concepts and reflect on the experiences of immigrant children by reading an excerpt of *Enrique's Journey*.

### 3 Informing curricular alignment through alternative assessments

Table 5: Proposal of activities for a lesson plan that integrates learners' self-assessment (via *Can-Do Statements*) in an Intermediate-level Spanish course for L2 and HL learners.


### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman


They are to analyze the reading, identifying how the content addresses issues of equity, diversity and inclusion. For the *Post-Reading video recording activity*, students are encouraged to use other sources of information and engage in an ethical debate about the nature and dangers of immigration. Students may discuss how *Enrique's Journey* parallels reality by referencing authentic newspaper articles of their choice. This activity subsequently allows students to become agents of their own knowledge.

In the *Closing activity*, students will discuss how their previous understanding on the topic has evolved from the preliminary activity (Blog 1). This final activity facilitates personal learning via the ongoing process of reflection and connection to human experiences and the realities of our globalized world. Teachers can tailor the lesson plan for L2 and HLLs by incorporating the questions in italics in the table and encouraging students to make personal connections to their own communities.

Throughout the sequence of proposed activities, students reflect upon their learning and monitor their own language development by means of the *Can-Do*

### 3 Informing curricular alignment through alternative assessments

*Statements* and self-assessment. Each section and activity thus measures to what extent learners feel comfortable to talk and write about diversity, equity, and inclusion as a framework for understanding Latinx immigration and border issues in the U.S. The goal of this proposed lesson plan was to provide language instructors with some direction on how to use self-assessment monitoring techniques to shape learners' perceptions of their language capabilities. As learners' metacognitive knowledge and learning strategies evolve, learners are better able to plan, carry out, and assess their own learning (Council of Europe 2002; Little & Perclova 2001).

### **5 Discussion**

One of the most challenging areas in language teaching concerns the scaffolding of student learning to transition from colloquial, everyday language use to a decontextualized, academic register. Cummins (1979) described this process within English language learners as the development of Basic Interpersonal Communication Skills (BICS) to advance toward Cognitive Academic Language Proficiency (CALP). For L2 learners, the development of BICS to CALP communication skills follows a rather predictable, linear pattern (Montrul 2011b). As evidenced in this study, however, such a linear trajectory does not necessarily exist for HLLs.

By virtue of their natural exposure to the language, HLLs often possess a wide array of linguistic skills and vocabulary, which typically manifest in a broader BICS communicative range than L2 learners. As a result, beginning HLLs may feel more confident with speaking the target language in social situations. This confidence may fluctuate, however, when HLLs are required to understand advanced grammatical concepts and apply this knowledge to speak or write in an academic register (Beaudrie 2009; Carreira 2003; Correa 2011; Montrul 2011a; Zyzik 2016). Given this variability in HLL self-efficacy, it is essential that selfreported assessments, such as the *Can-Do Statements*, serve as a flexible document through which instructors can better align their practices to the needs of their student population (Cox et al. 2018: 108).

The present study thus examined four levels of coursework to determine the development of BICS throughout Beginning and Intermediate levels. Of particular interest was exploring the bridging of BICS and CALP skills within the fourth semester course, Intermediate Spanish II. To explain, beginning-level coursework provides learners with opportunities, such as role-play exchanges, to learn the target language for sophisticated, social interactions. Knowledge of BICS is therefore important to help learners feel comfortable speaking and writing the language in socially and culturally appropriate contexts. On the other hand, knowledge of CALP is essential for academic success and for critical thinking.

The fourth semester of the last intermediate language course is therefore a pedagogically operationalized space where BICS and CALP can be included in cognitively stimulating and socially meaningful activities. In this regard, the incorporation of self-assessment of output language skills can help learners reflect upon their own language learning and identity in a more globalized society where different cultures and linguistic profiles are equitably represented. The set of classroom activities proposed in this study were thus designed with *equity* as a core principle.

Students enrolled in the Beginning and Intermediate courses in the SSL program developed their speaking and writing skills in Spanish through authentic media-based readings that focused on the social and political realities of Central American immigration to the United States. The inclusion of culturally representative readings, such as *Enrique's journey*, makes a case for a responsive pedagogy that is sensitive to the social realities of the Spanish language and culture, while also having students reflect on their self-efficacy. As such, the sample lesson plan provided in this chapter facilitates and enhances learner self-confidence through activities that promote a sense of belonging through the Latinx community, and validation of self-worth and cultural and linguistic identity.

Affirmative *Can-Do Statements* can contribute to the validation of self-worth and learner identity, as they intrinsically make learners reflect where they are in the process of learning of Spanish. As previously mentioned, self-assessment stimulates learners' autonomy and their language development. Through the implementation of self-assessment checklists identified in the *Can-Do Statements*, learners can develop specific skills such as monitoring, planning, improving and evaluating their own learning. Additionally, the development of these skills presents advantages that are valuable to the learners beyond the classroom setting. For instance, self-assessment helps learners to develop meta-cognitive skills that can be applied later on in their lives to evaluate their professional performance. It also enhances self-awareness through reflective practice. Likewise, by exercising self-assessment, language learners improve their critical reviewing skills facilitating them to be more objective of their own learning and others. Last but not the least, self-assessment enhances learners' agency by enriching their own learning through meaningful reflection and evaluation. As such, self-assessment is a valuable, alternative form of assessment to traditional assessments as long as students have been guided and trained on how to self-evaluate their own learning.

The empirical results reported herein should be considered in the light of some limitations. Foremost, the present study did not include the instructors' perceptions regarding the capabilities of their students, and as such, there is no com-

### 3 Informing curricular alignment through alternative assessments

parative data that explores the learner's self-perceived capabilities of Spanish speaking and writing with their instructors' observations. Another variable that was not investigated in this study was how language anxiety influenced learners' self-reported data. The role of anxiety in relation to different language profiles, such as second/heritage language learning, should subsequently be explored in the future. Additionally, communicative anxiety when writing and/or speaking any languages and the age of acquisition are other important factors to examine (Dewaele et al. 2008; Sparks & Ganschow 1991). Lastly, this study only focused on speaking and writing skills. Further research should therefore examine learners' self-perceived capabilities of language learning in other skills and competencies, such as listening, reading and intercultural competence; to see if self-efficacy traits are similar to or different from those found in speaking and writing.

### **6 Conclusion**

This study contributes to existing research on the exploration of self-efficacy within the context of higher education language learning classrooms. By examining how participants in two Spanish language programs self-assessed their abilities in two domains, written and spoken, the authors were able to design a pedagogical tool that allowed learners to advance their level of proficiency through inclusive and diverse classroom activities. Overall, the findings underscore the importance of measuring learners' perceived linguistic capabilities as a method to inform classroom instruction. Specifically, intermediate language coursework must incorporate tasks that require learners to routinely measure their selfefficacy. This increased awareness of one's linguistic capabilities is especially critical in mixed classrooms, where the differing levels of proficiency, diverse learner profiles, and self-efficacies can exacerbate learners' feelings of anxiety. By implementing an instructive design based on student self-efficacies and curriculum objectives, instructors are able to 1) help learners positively identify their language skill sets; and 2) better plan activities that encourage learners to build their self-efficacies in oral and written skills. This insight can then inform best practices that integrate language awareness and self-assessment through the lens of social justice. Alternative assessments that are based on self-efficacy driven *Can-Do Statements* can thus bring into language classrooms current events and cultural manifestations that are meaningful within Latinx communities.

In conclusion, the findings and proposed curriculum in the present study call for a more contextualized approach to language instruction and planning that takes into consideration the learning outcomes that the students' themselves identify as goals to their success. By integrating learners' voices on self-perceived capabilities into language coursework, instructors may draw on this kind of data as a baseline for the development of a more reliable set of course learning outcomes in both cross-sectional and vertical curriculum alignment.

### **References**


3 Informing curricular alignment through alternative assessments


### E. Rodríguez-González, M. Giráldez-Elizo & S. Schulman


3 Informing curricular alignment through alternative assessments


## **Chapter 4**

## **Measuring L2 pragmatic development in the study abroad context**

### Todd A. Hernández

Marquette University

Although it is often assumed that study abroad (SA) provides an ideal environment for second language (L2) pragmatic development, research has demonstrated that students' lack of consistent exposure to the target language imposes limitations on such development (Barron 2003, Kasper & Rose 2002, Shively 2010). Further, studies have found that even highly motivated SA are often unaware of how to take full advantage of the SA environment to strategically develop their pragmatic competence (Cohen & Shively 2007, Hernández 2016, Shively 2010). Given this background information, it is clear that programs must develop reliable models for assessing the effects of the SA experience on pragmatic development and other L2 outcomes as well.

I begin this chapter with an overview of pragmatic competence in the SA context. This chapter then focuses on methods of assessing L2 pragmatic development in SA environments. I review and provide examples of research that employs a wide range of assessment methods, from discourse completion tasks (DCTs) and controlled production tasks to role-plays and more open-ended production tasks and naturalistic data. In addition, retrospective verbal reports, language contact profiles, and journal entries are discussed as tools that can inform and improve SA participants' learning. I then discuss studies that have assessed the impact of pedagogical intervention on SA students' L2 pragmatic development. I conclude this chapter with a discussion of the implications of SA assessment of pragmatic development and outline directions for future research.

### **1 Introduction**

Pragmatic competence, a core feature of communicative competence and a central goal of foreign language instruction (Bachman & Palmer 1996, Canale &

Todd A. Hernández. 2022. Measuring L2 pragmatic development in the study abroad context. In Eva Rodríguez-González & Rosita L. Rivera (eds.), *Language assessment in multilingual settings: Innovative practices across formal and informal environments*, 67–103. Berlin: Language Science Press. DOI: 10.5281/zenodo.6762276

Swain 1980), refers to "one's knowledge of linguistics, norms, and social conventions, and one's ability to use these knowledge bases in a socially-bound interaction" (Taguchi 2015: 1). Second language (L2) learners interested in achieving native-like competence in a target language must acquire two distinct yet related knowledge bases: pragmalinguistic knowledge, or knowledge of the specific forms and linguistic strategies involved in the performance of a speech act (e.g., how to refuse an invitation), and sociopragmatic knowledge, or knowledge of the social norms of a culture and how they influence interactional patterns (Thomas 1983).

Study abroad (SA), a formal educational experience by a student in a target language country, is often considered an ideal context for L2 pragmatic development because it has the potential to provide SA learners with access to large amounts of input and interaction with native speakers (Hernández 2010, Ren 2018, Vande Berg et al. 2012). SA is "characterized by an uninstructed (i.e., implicit) component that may or may not combine with an instructed (i.e., explicit) component" (Sanz 2014: 1). More than in the foreign language classroom, L2 learners have frequent opportunities during SA to use the target language to perform a wide range of speech acts or communicative functions (e.g., apologies, compliments, requests, refusals) that have real-world consequences (Shively 2011). Although previous research has shown that SA has a positive impact on L2 pragmatic development for some learners, there is considerable individual variation in outcomes (e.g., Barron 2006, Bataller 2010, Félix-Brasdefer & Hasler-Barker 2015, Hernández 2018b, Shively & Cohen 2008). SA assessment research has identified several factors that contribute to this variation: "quantity and quality of contact with the L2, length of stay, living situation, density of L2 speaking social networks, and individual characteristics (e.g., proficiency, motivation, gender, age, identity, dispositions)" (Pérez Vidal & Shively 2019: 355). Learner agency has also been proposed as having an influence on L2 pragmatic development (LoCastro 2003, 2012). That is, SA learners may choose not to adopt target pragmatic norms when the norms contradict their social identity (e.g., Bataller 2010). Finally, the fact that SA students often do not receive corrective feedback from native speakers about pragmatics and are thus frequently unaware that their language use does not conform to host culture norms represents yet another challenge to L2 pragmatic development in the SA context (Hernández & Boero 2018b, Shively 2010).

This chapter offers an overview of studies that have measured L2 pragmatic development in the SA context. The first section discusses several assessment methods that have been employed to measure SA participants' pragmatic competence. A review of the existing literature on uninstructed and instructed L2

4 Measuring L2 pragmatic development in the study abroad context

pragmatic development follows. Some suggested avenues of future research are then given.

### **2 Data collection methods for assessing L2 pragmatic competence in SA**

Given the social nature of pragmatic competence, one of the challenges researchers face in measuring pragmatic knowledge concerns collecting data "that closely reflects L2 learners' language use in social contexts" (Taguchi 2018: 7). While experimental data are often criticized for their lack of authenticity, a disadvantage of naturalistic data is their lack of generalizability or comparability (Taguchi 2018, Taguchi & Roever 2017). Data collection methods commonly used to assess SA participants' L2 pragmatic competence include: written discourse completion tasks (DCTs) (e.g., Cohen & Shively 2007, Hernández 2016, 2018a, Hernández & Boero 2018a, Shively & Cohen 2008); oral DCTs (ODCTs; e.g., Félix-Brasdefer & Hasler-Barker 2015, Halenko 2018, Hernández 2021, Taguchi et al. 2016) and multimedia elicitation tasks (METs; e.g., Schauer 2004); role-plays (e.g., Bataller 2010, Hernández 2018b, Woodfield 2012); audio recordings of tasks performed abroad (e.g., Hernández & Boero 2018a, Morris 2017); journal entries (e.g., Dufon 1999, Hernández 2018b, Shively 2011); naturalistic data (e.g., Dufon 1999, Shively 2011, 2015); retrospective verbal reports (RVRs; e.g., Hernández 2018a, Ren 2014, Woodfield 2012); and native speaker perceptions of pragmatic appropriateness (e.g., Cohen & Shively 2007, Hernández 2016, 2018a, 2021, Hernández & Boero 2018a, Li 2014, Shively & Cohen 2008, Taguchi 2011).

DCTs, sometimes referred to as written production questionnaires, have been one of the most frequently used methods to measure L2 pragmatic development in the SA context (Félix-Brasdefer 2010, Taguchi 2018). Because DCTs are an indirect measure of pragmatic speaking ability, the data do not necessarily reflect what participants would actually say in a particular situation. Instead, this assessment method measures what they think that they would say (Golato 2003, Shively & Cohen 2008). Nevertheless, the DCT format offers several advantages compared to other similar measures, as outlined by Shively & Cohen (2008) and affirmed by other researchers:



Table 1 provides a description of the request scenarios that appeared in Shively & Cohen's (2008) DCT. As the authors indicate, each scenario captures social and situational variation based on three factors: social status, social distance, and degree of imposition. The scenarios themselves also provide important background information about the setting, topic, interlocutor relationships, and the goal of the interaction (Taguchi 2018).

Shively & Cohen's (2008) DCT used a multiple-rejoinder approach requiring the students to fill in the blanks of a dialogue that contained several responses from an interlocutor. The following is a sample request scenario adapted from their study:

Paper Extension: You find a great bargain airfare for this weekend only, which you want to make use of in order to visit good friends in a somewhat distant city. In order to take advantage of this deal, you need to ask your professor, Dr. Rodríguez, for an extension on a paper that you were going to work on this weekend, and which is due next week.

You:

Dr. Rodríguez: Mira, es que creo que ya tuviste mucho tiempo para trabajar en este proyecto durante el fin de semana. No deberías haber esperado hasta el último momento para terminarlo.

You:

Dr. Rodríguez: Lo siento, pero no puedo darte más tiempo para entregar este trabajo. No creo que ir a visitar a unos amigos sea una buena excusa para pedir más tiempo.

You:

Dr. Rodríguez: Bueno, la verdad es que no me gusta hacer este tipo de cosas.

### 4 Measuring L2 pragmatic development in the study abroad context

### You:

Dr. Rodríguez: Bueno, está bien, pero solamente esta vez.

Table 1: Description of Shively & Cohen's (2008) request scenarios on their DCT


Role-plays are the second assessment method. Similar to DCTs, role-plays allow for the manipulation of social variables while eliciting spoken data through simulated interactions between two or more interlocutors who act out predefined roles (Félix-Brasdefer 2010, Félix-Brasdefer & Hasler-Barker 2017, Kasper 2000). Considered a more valid assessment tool for measuring L2 learners' pragmatic competence than DCTs, this method has gained increasing interest because it allows participants to engage in real-time negotiation and interaction (Bataller 2010, Bataller & Shively 2011). In addition, intonation, repetition, pauses, listener responses, and turn-taking are important pragmatic features of natural speech that can be captured with role-play data (Turnbull 2001). In terms of use, the

participant reads a description of a situation and is asked to respond as if she or he were in that real situation. Role-plays consist of two types: closed and open (Kasper & Dahl 1991). In a closed role-play, the participant responds to a situation with one turn, and without a response from the interlocutor. In an open role-play, while the roles of the interlocutors are specified, the participant may use as many turns as he or she needs to complete the situation.

Although traditional DCTs provide a written description of the setting, the speakers, and the goal of the interaction, they have often been criticized for lacking the extralinguistic features (e.g., gestures or personal distance) that one would expect when performing a speech act in the real world (Félix-Brasdefer & Hasler-Barker 2017, Rockey et al. 2020). To address this issue, some researchers have designed computer-based ODCTs or METs that offer rich audiovisual and contextual information in the situation prompt (e.g., Culpeper et al. 2018, Félix-Brasdefer & Hasler-Barker 2017, Schauer 2004). As such, ODCTs better simulate authentic interactions than traditional DCTs or written production questionnaires (Félix-Brasdefer & Hasler-Barker 2015, Halenko 2018, Hernández 2021, Ren 2014, 2019, Taguchi et al. 2016). Schauer's (2004) MET, consisting of 16 scenarios focusing on requests, controlled the timing and nature of the audio and visual input provided to participants through a computer-based presentation format. Similarly, Schauer's (2009) MET controlled for interlocutor effects (e.g., tone of voice). In a UK-based SA program for ESL learners, Halenko (2018) employed an ODCT to assess the effects of pragmatic instruction on learners' ability to formulate English apologies. The scenarios featured a range of animated interlocutors and problems which the learners had to address by responding in single turn interactions. In a similar study, Hernández (2021) employed an ODCT consisting of five situations to measure the effects of a pedagogical intervention on SA participants' Spanish apologies. Figure 1 shows one of the ODCT scenarios from Hernández (2021). The scenario itself was adapted from Shively & Cohen's (2008) DCT.

Naturalistic audio recordings, the fourth assessment method, have also been employed to measure L2 pragmatic development in the SA context (Bataller & Shively 2011, Dings 2014, Dufon 1999, Shively 2011, 2013, 2015). Shively's (2011) SA participants made audio recordings of themselves while participating in service encounter interactions with service providers in Spain. In advocating for the use of naturalistic data for L2 pragmatics research, the author suggested that the service encounters analyzed had real-life interactional and psychological consequences. Similarly, according to Bataller & Shively (2011), naturally occurring data better capture how SA students interact in real-life encounters than do roleplays or production questionnaires. Although naturalistic data might represent

### 4 Measuring L2 pragmatic development in the study abroad context

Figure 1: ODCT scenario from Hernández (2021)

a more valid measure of SA students' pragmatic competence than other methods, several practice challenges exist in their application (e.g., comparability and unpredictability of data; time-consuming nature of data collection).

RVRs, the fifth assessment method, consist of obtaining verbal reports from a learner after completion of a task while information is still available in the learner's short-term memory (Félix-Brasdefer 2010). This procedure provides insights into the cognitive processes that learners employ while performing a task, such as information on their planning of a speech act, their language of thought, and their choice of language forms (Ren 2014, Woodfield 2012). RVRs are considered by some as essential for determining speech act performance because "one may learn what the respondents actually perceived about each situation (e.g., what they perceived about the relative role status of the interlocutors) and how their perceptions influenced their responses" (Cohen 2004: 321). Woodfield (2012) employed RVRs to investigate the perceptions of SA students with regard to their performance of two role-plays eliciting status-equal and status-unequal requests. The RVRs indicated that the students paid attention to linguistic form, sociopragmatic and pragmalinguistic knowledge, and politeness. The RVRs also suggested that it was often difficult for students to select appropriate forms to meet the goal of their interactions. Based on these findings, Woodfield concluded that RVRs help reveal learners' current sociopragmatic and pragmalinguistic knowledge while informing researchers of the linguistic difficulties that learners face during speech act production.

Shifting to the sixth assessment method, some researchers have also employed native speaker perceptions of appropriateness as a measure of L2 learners' pragmatic competence (e.g., Halenko 2018, Hernández 2016, 2018a, 2021, Hernández & Boero 2018a, Li 2014, Shively & Cohen 2008, Taguchi 2011). Taguchi (2006) defined pragmatic appropriateness as "the knowledge of the conventions of com-

munication in a society, as well as linguistic abilities that enable learners to communicate successfully" (2006: 513). Criteria often used to assess appropriateness have included aspects of language use, level of formality, directness, politeness, word choice, and grammar (Taguchi 2011). Table 2 shows the appropriateness rating scale for requests designed by Shively & Cohen (2008) and subsequently used by Hernández (2016, 2018a) and Hernández & Boero (2018a). The reader interested in the speech act of apologies is directed to Shively & Cohen (2008) for that rating scale.

> Table 2: Shively & Cohen's (2008) pragmatic appropriateness rating scale for requests


SA participants' journal entries, the seventh assessment method, have been used in several studies to investigate L2 learners' developing awareness of pragmatic norms (e.g., Dufon 1999, Hernández 2018b, Shively 2011). As the eighth and final assessment method described in this chapter, audio recordings of tasks performed during the SA program have the potential to provide formative feedback to SA students and thereupon enhance their language acquisition. In Hernández & Boero (2018a), participants were given tasks designed to strategically develop their pragmatic competence. Task performance was assessed by the researchers and explicit feedback was given to the L2 learners "to draw their attention to mismatches between their language use and pragmatic choices and those of the host culture" (2018a: 396). DCTs have been one of the most frequently used methods to assess L2 pragmatic development in SA contexts for their ease of administration and ability to manipulate contextual variables (Félix-Brasdefer & Hasler-Barker 2017, Rockey et al. 2020). To address some of the disadvantages of this instrument, researchers have begun to adopt other methods for evaluating pragmatic competence, such as role-plays, ODCTs, and naturalistic data, among others. Each also has its advantages and disadvantages. This chapter now shifts to a discussion of uninstructed and instructed L2 pragmatic development in SA.

4 Measuring L2 pragmatic development in the study abroad context

### **3 Uninstructed L2 pragmatic development in SA**

Although studies on the development of pragmatic competence in the SA context have tended to focus on the investigation of speech acts (Kasper & Dahl 1991, Ren 2018), other areas have also been researched: address forms (e.g., Dufon 1999, Hassall 2013, 2015a,b); conversational style (e.g., Cordella 1996); humor (Shively 2013); interactional competence (e.g., Dings 2014, Masuda 2011, Shively 2015, 2016, Taguchi 2014a); impoliteness (Félix-Brasdefer & Hasler-Barker 2017); conversational implicature (e.g., Taguchi 2008a,b); and pragmatic routines (e.g., Alcón-Soler & Sánchez Hernández 2017, Osuka 2014). Table 3 lists all existing research concerning uninstructed L2 pragmatic development during SA, published 1996 to 2021. Studies are organized by eight pragmatic features. Of these, the eighth, speech acts, is further subdivided into ten individual speech acts. For each study, the table presents author information (in alphabetical order), the target language, assessment methods employed by the researcher(s), and the duration of the SA program.


Table 3: Studies on uninstructed pragmatic development in SA


### 4 Measuring L2 pragmatic development in the study abroad context




*<sup>a</sup>* While Schauer (2006, 2009) focused on perception of appropriateness of requests in English, apologies, refusals, and suggestions were also investigated.

### **4 Development of receptive pragmatic competence in SA**

This section provides an overview of studies that have investigated L2 learners' development of receptive pragmatic competence in SA programs. SA participants made gains in their comprehension of leave-taking expressions (e.g., *Au revoir.* 'Goodbye') and address forms in French (Kinginger 2008), recognition of routine pragmatic formulae in English (Alcón-Soler & Sánchez Hernández 2017, Roever 2012) and Japanese (Osuka 2014), and perception of speech acts in English (Vilar Beltrán 2014, Schauer 2006, 2009). In addition, increased pragmatic awareness regarding Spanish requests has also been observed (Czerwionka & Cuza 2017b, Rodríguez 2001). Rodríguez's (2001) investigation is particularly interesting because the author compared SA students with "at-home" learners over the course of a semester. Although data collected from a judgment task and RVRs demonstrated that both groups of learners improved their receptive ability of requests, no significant differences were found between the two groups. In another study on the acquisition of receptive skills, Taguchi (2008a,b) found that ESL learners' gains in speed of comprehension of conversational implicatures over five months abroad were associated with self-reported language contact. Similarly, in Matsumura's (2001) study, the improvements ESL learners made in their choice of appropriate advice-giving expressions were attributed to the amount of selfreported exposure to English. Meanwhile, Bardovi-Harlig & Bastos (2011) found

that self-reported language contact had a positive effect on learners' recognition and production of conventional expressions in L2 English. Length of residence, on the other hand, had no effect. The authors concluded that the quality of social contact while abroad was more important than length of residence when it comes to the development of conventional expressions.

### **5 Development of productive pragmatic competence in SA**

As shown in Table 4, requests are the most investigated speech act in research on uninstructed L2 pragmatic development during SA (e.g., Bataller 2010, Han 2005, Hernández 2016, Shively & Cohen 2008). Researchers have examined other speech acts as well (e.g., Félix-Brasdefer 2004, 2013, Hernández 2018a, Hoffman-Hicks 1999, Schauer 2006, Schauer 2009). Employing the Cross-Cultural Speech Act Realization Project Coding Manual (Blum-Kulka et al. 1989) and Brown & Levinson's (1987) politeness theory as frameworks, several researchers have looked at how SA participants use (in)direct strategies, internal and external mitigation, formulas, semantic strategies, and deictic orientation in their speech acts before and after the SA program.

Regarding apologies, some researchers have reported gains during SA (Warga & Schölmberger 2007), whereas others have found little development of this particular speech act (e.g., Hernández 2018a, Kondo 1997, Shively & Cohen 2008). One finding is conclusive: consistent with what has been observed for other areas of L2 learners' pragmatic competence, the development of SA participants' apologies is subject to significant individual variation. Using a DCT, Kondo (1997) measured the development of L2 English apologies by Japanese high school learners before and after their academic year in the United States. The findings indicated that the participants became more target-like post-SA by shifting from overuse of Expressions of Apology (e.g., 'sorry') to greater use of Explanations (e.g., I was late because there was an accident). Other forms of mitigation were also increased. Hernández (2018a) found that 18 SA students did not improve several features of their Spanish apologies after short-term (4 weeks) SA. Students overused the expression of apology or illocutionary force indicating device (IFID), *lo siento* ('I'm sorry'), on pre- and posttest DCTs. In contrast, they employed IFID Intensification (e.g., Lo *siento de verdad.* 'I'm truly sorry') and the agentless construction in the Acknowledgement of Responsibility strategy (e.g*., Se me olvidó …* 'I forgot') less frequently than the native speakers. Hernández concluded that exposure to target language input during SA may well be insufficient for participants "who have the expectation of acquiring the pragmatic features of the host community"

### 4 Measuring L2 pragmatic development in the study abroad context

(2018a: 616). As such, the author argued that SA programs should incorporate explicit information about pragmatics into pre-departure orientation and again over the course of the experience abroad. Given the results of this study and others, pragmatic instruction might be particularly beneficial to SA participants in short-term programs.

SA research indicates that L2 pragmatic development does not always occur in a linear fashion, as Warga & Schölmberger's (2007) study of seven Austrian learners of French who studied for ten months in Quebec demonstrated. Three major developmental patterns were observed. First, SA participants became more target-like by reducing their use of Excuses and Justifications. Contrastively, the second development consisted of several shifts in the opposite direction of the target norm (e.g., increase in the use of two upgraders in one IFID). Third, during their program abroad the SA learners did not change their frequency of use of IFIDs, which they employed more frequently than the native speakers. An additional finding was the participants' overuse of *malheureusement* ('unfortunately') before SA. The authors suggested that they had transferred this strategy from their L1. During the second and third data collection times, the students decreased their use of *malheureusement* and replaced it with target language chunks. By the fourth time, learners had begun to replace these target-like chunks with a more controlled and creative pragmatic performance. At this stage, the SA participants had combined native-like strategies with elements from their L1. The authors concluded that at the final stage of acquisition, the L2 learners would have target-like control of this feature. Employing corpus-based methods, Barron (2019) investigated the development of L2 apologies by SA participants who spent ten months in Germany. The overuse of explicit IFIDs and single routine chunks were two of the pragmatic features that remained non-target-like over time.

Turning to L2 request development, studies have found that SA participants improve in some aspects of their request behavior, as other features remain the same (e.g., Barron 2003, Barron 2006, Bataller 2010, Cole & Anderson 2001, Han 2005, Shively & Cohen 2008). In their investigation of the requests and apologies of 67 American SA participants over the course of a semester, Shively & Cohen (2008) found that although students improved their pragmatic appropriateness in some speech act scenarios, their formulation of requests and apologies was often non-target-like and at times even inappropriate. Two factors had a significant impact on specific speech act improvement: (1) the amount of time an SA participant spent speaking the target language outside of class with a fluent speaker of the L2, and (2) having extended conversations with the host family. Bataller (2010) examined 31 US students' requests in service encounters in a semester program

in Spain. Results showed that SA participants overused non-target-like forms on the pre- and posttests, such as query permission (e.g., *¿puedo tener una Coca-Cola?* 'can I have a Coke?'), need statements (e.g., *necesito* 'I need'), unmitigated direct requests (e.g. *quiero comprar otros zapatos* 'I want to buy other shoes'). In addition, they did not increase their use of simple interrogatives (e.g., *¿me pones un café?* 'will you give me a coffee?') or mitigated indirect request forms (e.g., *quería cambiar estos zapatos* 'I wanted to exchange these shoes'), which were the native Spanish speakers' preferred strategies. Based on these findings, Bataller concluded that pragmatic instruction should be explicitly taught in SA programs so that students be made aware of the target pragmatic features and norms of the host culture.

Using an ODCT, Li (2014) examined how language proficiency influenced the development of pragmatic knowledge, as measured by appropriateness ratings, and processing ability, as measured by planning time and speech rate. In this study, American learners in both an Intermediate and Advanced group were rated as pragmatically more appropriate in their L2 Chinese request production after a semester sojourn abroad. Neither group reduced planning time. The Advanced group improved their speech rate, whereas the Intermediate group did not. Findings suggest that the broader pragmatic and linguistic knowledge base of the Advanced group helped them take advantage of the SA environment and thus continue to develop their processing ability and pragmatic knowledge. In a study on request development during short-term SA, Hernández (2016) found that SA participants' use of verbal downgrading (e.g., conditional or imperfect to express politeness or hedging) and external modification (e.g., explanations or justifications for a request) did not change from pre- to posttest. In addition, participants overused speaker-oriented forms (e.g., *¿Puedo tener …?* 'Can I have?') both before and after SA – a phenomenon which researchers suggest is due to L1 transfer given that speaker-oriented requests are the norm in English (Félix-Brasdefer 2007, Márquez Reiter 2000, 2002, Pinto 2005). In his study, Hernández also measured the relationship between SA students' amount of language contact and their request development. No significant relationships were found, however. The author concluded that traditional short-term SA experiences might not provide participants with adequate opportunities for practicing the target language in the host community. Similarly, Taguchi's (2011) study on L2 English requests suggests that pragmatic gains made in SA may not be retained by L2 learners in the long-term.

Refusals are another speech act that have received substantial attention in the SA literature (Barron 2003, 2007, Félix-Brasdefer 2004, 2013, Ren 2014). Employing role plays, Félix-Brasdefer (2004) examined the relationship between SA par-

### 4 Measuring L2 pragmatic development in the study abroad context

ticipants' development of Spanish refusals and their length of stay in the target language country. He found that learners with longer SA experiences (30 months abroad) had stronger control of mitigation in refusal sequences than those learners who had spent less time abroad. In a similar study, SA participants did not significantly improve their Spanish refusals after a much shorter program of eight weeks (Félix-Brasdefer 2013). Ren (2014) investigated the cognitive processes of advanced English language learners studying abroad for one academic year. Participants were given eight ODCT scenarios eliciting statusequal and status-unequal refusals in English. The results of the RVRs showed that, over time, learners paid increasingly more attention to sociopragmatics in context while increasing their pragmatic knowledge.

Employing an ODCT, Félix-Brasdefer & Hasler-Barker (2015) compared SA participants' production of compliments in Spanish during an eight-week summer program in Mexico with students enrolled in an "at-home" context. Compliments involved four situations with differing degrees of social distance and power. The SA group showed some change with regard to one of the seven compliment strategies: *Qué* ADJ/ADV NP ('What ADJ/ADV NP'). In addition, the SA participants more frequently employed *padre* ('cool'), an adjective type often used when making compliments in Mexican Spanish. In contrast, the AH group did not show significant change from pre- to posttest. Both groups overused *Me* [*gusta*, *encanta*] NP (I ['like, love'] NP). The authors attributed the L2 learners' overreliance on this structure to three factors: (1) transfer (the English equivalent of this structure is frequent in English), (2) instruction (the dative Spanish structure is emphasized in the language curriculum), and (3) Andersen's (1990) Oneto-One Principle, which suggests that in the initial stages of language acquisition L2 learners tend to associate a structure with one function. Based on their findings, Félix-Brasdefer & Hasler-Barker concluded that "foreign language teaching as it stands is not sufficient to prepare learners to improve their pragmatic knowledge in specific areas, such as speech acts and other interactional routines. Although incidental learning in an SA setting helps improve the learner's pragmatic competence, instruction in pragmatics before and during SA maximizes their learning experience. That is, without specific focus on both pragmalinguistic and sociopragmatic information and exposure to relevant pragmatic input, learners are not prepared to approach appropriate interactions with NS in the target language context" (2015: 85).

Despite their importance for an L2 learners' communicative competence, conversational style, address forms, and humor are three underrepresented areas in the SA literature (Cordella 1996, Iwasaki 2008, Pérez Vidal & Shively 2019, Shively 2011). Cordella (1996) found that SA students who had spent time abroad acquired

a confrontational style of speaking (e.g., cooperative overlap, challenging questions, and interruptions) similar to that of the native Spanish speakers. Hassall (2013, 2015b,a) examined the development of Indonesian address terms by Australian SA participants during a summer program abroad. Although the learners acquired new address terms for the vocative slot, their knowledge of terms for the pronoun slot increased only modestly. The author concluded that the SA students would have benefitted from pre-departure pragmatic instruction about address terms with opportunities to notice, reflect on, and discuss the L2 norms for their usage. With regard to L2 humor, Shively (2013) followed an SA learner who developed his ability to be funny through participation in social interaction and feedback with age peers during one semester in Spain. Other studies (Bell et al. 2014, Kinginger 2015) have also found a positive effect for SA on L2 humor development.

Several studies have documented how L2 learners develop their interactional competence during SA (e.g., Dings 2014, Ishida 2010, Masuda 2011, Shively 2015, Shively 2016, Taguchi 2014a). Taguchi (2014a) investigated the development of interactional competence by L2 learners of Japanese during their semester abroad in Japan. She found that over time her 18 SA participants increased their use of incomplete sentences, an important interactional resource in Japanese conversations because of its function in the joint construction of talk. Shively (2015) examined listener responses (LRs) in Spanish during SA. She defined LRs as brief verbal responses (e.g., *claro* 'right,' 'of course', *ya* 'right') employed by a listener to provide feedback or support to the speaker, but which do not represent an attempt to take the floor or control topic development. LRs perform a wide range of functions essential for shaping ongoing discourse and developing intersubjectivity with the speaker: acknowledgement, understanding, receipt of previous talk, agreement, and evaluation. An analysis of 8,310 LRs revealed that over time the SA students shifted their use of speaker and listener responses in conversations with their native Spanish speaker interlocutors. However, because the SA group did not adopt allo-repetitions (e.g., repetition of all or part of a previous utterance) as a LR – an important interactional resource more frequent in Spanish than in English – Shively suggested that learners' attention should be drawn to this particular strategy and other LR tokens through pedagogical intervention.

Taguchi et al. (2016) examined the effects intercultural competence and amount of social contact had on the development of pragmatic knowledge by 109 American college learners of Chinese participating in a semester program in Beijing. An ODCT consisting of 24 scenarios, a social contact questionnaire, and the Cross-Cultural Adaptability Inventory (Kelley & Meyers 1995) were administered to participants pre- and post-program. Several important findings emerged from

### 4 Measuring L2 pragmatic development in the study abroad context

their analysis: First, the SA learners improved their speech act scores from pre- to posttest. Second, cross-cultural adaptability and social contact had a significant impact on pragmatic development. These findings are noteworthy because they suggest that to some extent SA participants' individual characteristics (e.g., in this case their intercultural competence) influence their access to opportunities for language practice and subsequent increases in pragmatic knowledge, thus, accounting for some of the variation observed in SA pragmatic outcomes research.

### **6 Instructed L2 pragmatic development in SA**

Studies on uninstructed L2 pragmatic development in SA environments have demonstrated that some students make progress in acquiring the pragmatic norms of the host culture (e.g., Shively & Cohen 2008). At the same time however, not all L2 learners improve their pragmatic competence (e.g., Bataller 2010, Hernández 2018a, Shively & Cohen 2008). Even fewer acquire native-like pragmatic ability. Social contact is one of several factors that has been identified as contributing to SA participants' pragmatic development. Length of stay represents another important factor that may affect acquisition. SA students in shortterm programs are at a potential disadvantage because of the inadequate exposure to the target language. Exacerbating these disadvantages, researchers have observed that even highly motivated students are often unaware of how to take full advantage of the SA environment for developing their pragmatic competence (Cohen & Shively 2007, Hernández & Boero 2018a, Shively 2010). As a result, in order to maximize outcomes, some researchers have advocated for the integration of classroom-based pragmatic instruction into SA programs because explicit instruction has been found to promote learners' pragmatic development (Martínez-Flor & Usó-Juan 2006).

Pragmatics instruction in SA contexts has tended to employ an awarenessraising approach to developing pragmatic competence (Halenko & Jones 2017, Hernández & Boero 2018a,b, 2019, Martínez-Flor & Usó-Juan 2006, Shively 2010). According to Schmidt (2001), there must be conscious noticing of a given target feature in the input for acquisition to occur. In the case of pragmatics, L2 learners must notice and understand not only linguistic form, but also pragmalinguistic function (Morris 2017). Because previous studies have shown that left to their own devices SA participants often lack the strategies to take full advantage of the SA context for developing their pragmatic skills (Cohen & Shively 2007, Hernández 2016, 2018a, Pérez Vidal & Shively 2019, Shively 2010, Shively & Cohen 2008), researchers have begun to investigate the effects of pre-departure

and in-country pedagogical intervention on L2 learners' pragmatic development. Table 4 lists all extant research assessing the impact of pedagogical intervention on SA participants' pragmatic competence, published between 1996 and 2020. Studies are organized by three pragmatic features. The third feature, speech acts, is thereupon subdivided into four individual speech acts. For each study, the table presents author information (in alphabetical order), the target language, assessment methods employed by the researcher(s), and the duration of the SA program.

### **7 Instructed pragmatic development in SA: Development of receptive pragmatic competence**

To date only two empirical studies that the author is aware of have investigated the effects of explicit pragmatic instruction on developing receptive skills of students in the SA context (Bouton 1999, Henery 2015). Consisting of six hours of instruction, Bouton's (1999) pedagogical intervention was effective in helping SA participants acquire conversational implicatures in English. Henery (2015) found that SA students who received guidance from an instructor developed their metapragmatic awareness of French (i.e., address forms and understanding of contextual factors that inform language choices) more than those participants who did not receive guidance. Her findings suggest that SA programs should encourage students to make linguistic observations during their time abroad while an expert instructor guides and scaffolds their interpretations.

### **8 Instructed pragmatic development in SA: Development of productive pragmatic competence**

Existing research assessing the impact of pragmatics instruction on SA students' productive pragmatic skills suggests that pragmatics instruction is beneficial (e.g., Cohen & Shively 2007, DiBartolomeo et al. 2019, Halenko 2018, Hernández & Boero 2018a,b, Morris 2017, Shively 2011). Employing DCTs, Cohen & Shively (2007) examined the impact of pedagogical intervention on 86 US students' acquisition of requests and apologies during a semester abroad in a French or Spanish-speaking country. In their pre-departure orientation, the experimental group received presentation, discussion, and practice activities on learning how to perform speech acts in the target language. Participants responded to practice scenarios, and then compared their answers to those of their peers. Next,

### 4 Measuring L2 pragmatic development in the study abroad context


Table 4: Studies on instructed pragmatic development in SA

*<sup>a</sup>*Morris's (2017) DCT focused on requests. Greetings, invitations, closings, and refusals were also measured.

the students compared their responses to native speakers of French or Spanish who had also completed the same speech act scenarios. While abroad, the experimental group wrote reflective e-mail journals about their language learning during the SA experience. The control group did not receive instruction on speech acts nor did they write journal entries. Although both groups increased their use of target-like speech act strategies, as measured by a production questionnaire, there was no significant effect for the intervention. Based on these findings, Cohen and Shively concluded that pragmatics instruction should consist of a more extensive intervention during pre-departure orientation. While abroad, SA participants should also be given practice activities that encourage noticing and reflection on pragmatic features.

In a study employing naturalistic data, Shively (2011) reported on the service encounter requests of seven SA students who spent a semester in Spain. A total of 131 audio recordings between the SA participants and service providers in Spain were analyzed. Results showed that the students shifted from speaker-oriented requests (e.g., *¿puedo tener un café?* 'Can I have a coffee?') to greater use of hearer oriented (*¿me pones un café?* 'Will you give me a coffee?') and elliptical forms (e.g., *un café, por favor* 'A coffee, please'). In their journal entries and in interviews with the researcher, the SA learners attributed their improvement to the explicit information about pragmatics they received during the first and fifth weeks of the semester and to socialization with the host community.

Halenko & Jones (2017) assessed the impact of pre-departure instruction on SA participants' production of requests by Chinese learners of English. Requests were measured by an ODCT administered as a pretest, posttest, and delayed posttest. Findings indicated that explicit instruction had a significant effect on the immediate posttests. Significant attrition was found on the delayed posttests, however. The authors concluded that regular, repeated metapragmatic instruction might have had a more positive impact on SA students' acquisition of pragmatic features than the practice given to them within a short time period.

In a similar study, Halenko (2018) investigated the efficacy of an ODCT for assessing and improving the spoken communication skills of Chinese learners of ESL during their UK-based SA program. ESL learners were assigned to either an experimental group who received computer-assisted explicit instruction on apologies, or a control group that did not receive instruction. The instruction followed Usó-Juan's (2010) five stages of awareness-raising and communicative practice. In stage one, learners were introduced to the linguistic and cultural aspects of the speech act of apologizing. During stage two, they were presented with the formulaic expressions used to offer apologies. In the third stage, the researcher provided the learners with authentic scenarios designed to encourage

### 4 Measuring L2 pragmatic development in the study abroad context

reflection on aspects associated with Brown & Levinson's (1987) politeness theory, such as power, distance, and imposition – and their relationship to speech act production. ESL learners then engaged in communicative practice in stage four, followed by teacher-led discussion and feedback in stage five. ODCT data showed that computer-assisted explicit instruction and practice was indeed an effective tool for developing and assessing the SA participants' pragmatic development of L2 apologies.

Morris (2017) examined the effects of a task-based intervention on the pragmatic development of twelve SA participants during a ten-week program in Spain. Pragmatic development was measured by pre- and posttest DCTs, audio recordings of tasks performed in the target culture community, and selfreflections of task performance. Participants demonstrated significant gains in pragmatic competence (e.g., giving directions and ordering meals in Spanish) on the DCT and in their task completion. Similarly, their reflections showed increased awareness of the pragmatic features introduced to them during the entire pedagogical intervention. Using an ODCT and journaling, Winke & Teng (2010) measured the impact of task-based pragmatics instruction during an SA program in China. SA participants improved their use of formulaic expressions in speech act production. In addition, students' journal entries revealed developing awareness of Chinese pragmatics.

Hernández & Boero (2018a) found a positive effect for pragmatics instruction in the short-term SA context. Prior to departure, 15 SA participants received explicit instruction on requests, then guided practice with authentic input and awareness-raising activities, output practice, and subsequent group discussion and feedback. In Spain, they performed four simulated tasks that required them to make requests. The tasks varied in terms of social distance, social status, and degree of imposition. Next, the SA participants responded to questions that encouraged reflection about the appropriateness of their request strategies. Feedback was also given. In addition to significant gains on posttest DCTs, RVRs demonstrated that specific sociopragmatic factors and pragmalinguistic strategies targeted during the intervention informed learners' cognition during speech act planning and production.

In another investigation on short-term SA, Hernández & Boero (2019) employed DCTs to measure participants' request making in two service encounter scenarios. The first scenario was ordering a drink at a café. The second involved exchanging a pair of shoes without having the receipt. The SA participants made significant gains in both scenarios. Hernández & Boero (2018b) found that the positive effects of their pragmatic intervention extended to students' request production in role-plays. In addition, most of the gains were retained in delayed

posttests administered five weeks after the conclusion of the SA program. The authors attributed the SA participants' gains in the long term to the repeated exposure to pragmatics instruction that the L2 learners received during their entire time abroad. Employing role-plays, Hernández (2018b) examined the effects of pedagogical intervention on SA participants' requests during a six-week program in Buenos Aires, Argentina. Findings again demonstrated that explicit pragmatic instruction combined with in-country pragmatic data gathering tasks contributed to the students' L2 pragmatic development.

Russell & Vásquez (2018) examined the impact of a web-based tutorial (WBT) on 13 US students' pragmatic development during the semester prior to studying abroad in Spain. Findings indicated that the SA participants slightly increased their use of pragmatically appropriate strategies on the posttest production questionnaire, especially in the area of conventional politeness formulae to frame a complaint. Pre- and posttest scores on a comprehension-based assessment revealed that their ability to comprehend or notice appropriate pragmatic strategy use in Spanish also improved. The authors concluded that instruction on specific grammatical features taught in context prior to the WBT – such as the past subjunctive and conditional mood in Spanish – may help students formulate more appropriate hedges and downgraders in their speech act production. DiBartolomeo et al. (2019) reported on SA students' compliments and apologies after five weeks in Mexico. Over the course of their time abroad, participants received pragmatic instruction on compliments but not on apologies. Most of the SA learners improved their use of compliments. In contrast, few students adopted the native norms for apologies in Spanish. The authors concluded that the pragmatic instruction was responsible for the differential outcomes observed at the time of the posttests.

Hernández (2021) considered multiple measures of pragmatic knowledge in his investigation on the effects of pedagogical intervention and short-term SA on the development of L2 Spanish apologies. Students who spent four weeks in different SA programs in Spain were randomly assigned to an experimental group (*n* = 9) or a control group (*n* = 9). Prior to departure, the experimental group received explicit pragmatic instruction on how to formulate apologies. During SA, they performed two task scenarios designed to improve their pragmatic competence. The control group did not receive explicit pragmatic instruction, nor did they perform the tasks. L2 pragmatic development was assessed by an ODCT consisting of five apology scenarios. The ODCT data demonstrate that the experimental group significantly outperformed the control group in three measures of pragmatic development: appropriateness ratings by native speakers, strategy

### 4 Measuring L2 pragmatic development in the study abroad context

use, and speech rate. The agentless construction in the Acknowledgement of Responsibility strategy (e.g., *Se me cayó* 'I dropped') and IFID Intensification (e.g., *Lo siento de verdad.* 'I'm truly sorry'), two problematic structures for L2 Spanish learners, were among the features that the experimental group increased on the posttests. The experimental group also reduced their use of the single chunk *lo siento* 'I'm sorry' while expanding their IFID repertoire (e.g., *Perdóneme.* 'Excuse me'). In contrast, the control group did not improve their apologies during SA, often overusing routine expressions.

Mir (2020) examined the impact of pedagogical intervention on SA participants' L2 Spanish compliments during a four-week program in Spain. Data were collected by an ODCT consisting of eight complimenting scenarios. Over the course of the first two weeks, 20 students received explicit instruction on Spanish pragmatics, with a focus on speech act of compliments and compliment responses. Next, they were asked to conduct ethnographic research in the host community. Activities included: (1) asking their host families to perform several role-plays, which the students recorded and thereupon transcribed; (2) recording, transcribing, and analyzing conversations in the host community; and (3) practice employing compliments and compliment responses with host family members. For all three tasks, the SA participants shared the information they gathered from the native informants in subsequent class discussions focusing on pragmatics. Weekly journal entries were also employed to provide learners with further opportunities to discuss and interpret the pragmatic data. The ODCT data show that the explicit classroom-based pragmatic instruction combined with the ethnographic work helped the SA participants develop target-like complimenting strategies (e.g., *Que* + ADJ/NP construction) on the posttests.

Although studies on instructed L2 pragmatic development have predominantly focused on pragmatic speaking ability, Alcón-Soler (2015) conducted a unique study on the assessment of writing skills in L2 English. Specifically, the author investigated the impact of pragmatics instruction on learners' ability to mitigate requests in e-mail communication with their instructors. The experimental group received instruction on e-mail requests in which their attention was drawn to request forms, sociocultural norms, and discourse structure in academic e-mail communication. In addition, this group was provided with practice and feedback. The control group did not receive instruction but did exchange emails with their instructor. Although the quantitative analysis showed that both groups improved on the posttests, the qualitative analysis revealed that the experimental group had a more developed understanding of the relationship between forms and context.

Shively (2010) provided the most comprehensive model to date for teaching pragmatics prior to, during, and after SA. During pre-departure, SA participants learn the relevance of pragmatics for SA. The primary goal of this stage is to build the L2 learners' confidence in the target language so that they can maximize their SA experience. Participants' attention is drawn to the pragmatic norms (e.g., norms about interaction in the target language) of the host culture. Communicative practice with speech acts is another core component of Shively's predeparture stage. While abroad, SA students are encouraged to notice, reflect, and consider pragmatic features in out-of-class data gathering activities and with the guidance of an instructor or program director. The post-SA component of this model involves facilitating former SA students' "social interaction in the target language and continued development of the skills acquired in both the pre-departure and in-country stages" (2010: 123).

### **9 Conclusions and future directions**

In this chapter, I have provided an overview of studies that have measured L2 pragmatic development in SA contexts. First, I discussed the major assessment methods that have been employed to measure pragmatic knowledge in the SA environment. I then reviewed the existing scholarship on SA participants' acquisition of pragmatic competence in both uninstructed and instructed environments. The SA literature has demonstrated that some L2 learners improve some aspects of their pragmatic competence while participating in a sojourn abroad of a semester or more. Other pragmatic features develop slowly or are difficult to acquire through exposure alone. One important insight gained from assessment on SA students' L2 pragmatic development is that pedagogical intervention before and during SA has the potential to enhance pragmatic learning. Furthermore, as participation in short-term SA increases (Institute of International Education 2017), more studies should assess the impact that these programs have on participants' L2 pragmatic ability. Research suggests that students in these programs would benefit from instruction on pragmatic features during pre-departure orientation and subsequent opportunities to notice, practice, reflect on, and receive feedback about speech patterns during the SA program (e.g., Hernández 2021, Hernández & Boero 2018a,b).

With respect to assessment methods, researchers have begun to shift away from written production questionnaires or DCTs, which are perceived as having lower validity compared to other measures of pragmatic competence (Félix-Brasdefer 2010, Pérez Vidal & Shively 2019). ODCTs and role-plays are two such

### 4 Measuring L2 pragmatic development in the study abroad context

measures that may be better suited for capturing L2 learners' language use in social contexts. Although Rockey et al. (2020) did not focus on SA participants' pragmatic competence, their study on technology-enhanced (TE) DCTs demonstrates how researchers might employ a mobile application to increase the validity of traditional DCTs in SA contexts as well. In their study, the use of a TE-DCT accessed by a phone application provided students not only with important linguistic and extralinguistic cues, but it also allowed them to employ these same features in their own responses. Based on this data, the authors argued that the use of mobile applications was an ecologically valid way to measure one type of pragmatic ability. In addition to issues surrounding increased face and content validity, TE-DCTs would also enhance researchers' ability to collect L2 pragmatic data from SA participants in different programs. RVRs also offer a potentially rich source of data about how SA learners perceive social and cultural variables associated with different speech acts. Naturalistic data, for their part, allows programs to investigate how SA participants make use of their pragmatic skills in authentic situations. Asking SA participants to become data gatherers is one approach to obtaining authentic interactional data (e.g., Shively 2011). A second approach is to analyze how students use the target language in social media and chat applications such as Facebook, Instagram, or WhatsApp (Pérez Vidal & Shively 2019).

In terms of directions for future research, a worthwhile avenue of investigation would be to use a simulated 3-D SA environment (e.g., with actors from the target language country and simulated situations and surroundings) as pre-departure practice, and also as a method for measuring participants' L2 pragmatic competence before and after an SA program. Given the increasing interest in SA programs with service learning, internships, volunteering, and work placement (e.g., an SA program designed for nurses; Engineers without Borders), research in this area is especially relevant. Similarly, researchers could combine this assessment method with measurements of stress or situational anxiety – an approach that has been identified in recent discussions about pragmatics in the SA context (Pérez Vidal & Shively 2019).

Most of the research on uninstructed L2 pragmatic development in SA has focused on speech acts (e.g., Félix-Brasdefer & Hasler-Barker 2015, Hernández 2016, 2018b, Ren 2014, 2019, Shively & Cohen 2008). Although these studies have contributed important information about what SA students can and cannot acquire during their time abroad, other aspects of pragmatics should also be investigated. L2 learners' interactional competence (e.g., discursive abilities, discourse marker use, backchanneling) and pragmatic processing ability (e.g., Hernández 2021, Li 2014) are two areas that merit further attention.

It should also be noted that the difficulties of measuring L2 pragmatic competence are exacerbated by the fact that some L2 learners may consciously use a non-target-like feature because they know it is correct even though it is not pragmatically appropriate. Barron (2003)referred to this phenomenon as the "playing it safe" strategy. Finally, language contact and motivation are often identified as important predictors of language acquisition in the SA context. However, with the exception of a few studies (e.g., Bardovi-Harlig & Bastos 2011, Hernández 2016, Matsumura 2001, Shively & Cohen 2008, Taguchi 2008a,b), research on how students' self-reported language contact or social networks influence L2 pragmatic development is underrepresented. Research that measures how individual differences influence the path of L2 pragmatic development that an SA learner takes in acquiring pragmatic competence during SA represents another important line of investigation.

The positive effects of pragmatics instruction on SA participants' developing pragmatic competence are promising. The focus of these studies has also been on speech acts, however. More research is needed on the impact of explicit classroom-based pragmatic instruction on other pragmatic features as well. Further, few studies have examined the impact of an intervention that takes place both before and during SA (e.g., Hernández 2021, Hernández & Boero 2018a, Shively 2011). Given that these interventions seem to be more beneficial for SA students' pragmatic development than pre-departure-only instruction, additional studies are justified. At the same time, further research on pedagogical interventions that encompasses not only these two important SA stages (pre-departure and in-country), but also examines post-SA pragmatic development is also warranted. Finally, assessment of L2 pragmatic development in SA has demonstrated that the "at-home" foreign language curriculum does not adequately prepare L2 learners to improve "their pragmatic knowledge in specific areas, such as speech acts and other interactional routines" (Félix-Brasdefer & Hasler-Barker 2015: 85). A language curriculum that targets the development of learners' pragmatic competence well before SA and then continues to build on this knowledge in predeparture and during the program itself will help SA participants maximize their learning experience. Systematic assessment of SA outcomes will undoubtedly assist with this effort.

### **References**

Alcón-Soler, Eva & Ariadna Sánchez Hernández. 2017. Learning pragmatic routines during study abroad: A focus on proficiency and type of routine. *Atlantis* 39(2). 191–210.




Rodríguez, Silvia. 2001. *The perception of requests in Spanish by instructed learners of Spanish in the second and foreign-language contexts: A longitudinal study of acquisition patterns*. Bloomington: Indiana University. (Doctoral dissertation).

Roever, Carsten. 2012. What learners get for free: Learning routine formulae in ESL and EFL environments. *ELT Journal* 66. 10–21.



Warga, Muriel & Ursula Schölmberger. 2007. The acquisition of French apologetic behavior in a study abroad context. *Intercultural Pragmatics* 4. 221–252.

Winke, Paula M. & Chunhong Teng. 2010. Using task-based pragmatics tutorials while studying abroad in China. *Intercultural Pragmatics* 7. 363–399.

Woodfield, Helen. 2012. Pragmatic variation in learner perception: The role of retrospective verbal report in L2 speech act research. In J. César Félix-Brasdefer & Dale A. Koike (eds.), *Pragmaticvariation in first and second language contexts: Methodological issues*, 209–223. Amsterdam: John Benjamins.

## **Chapter 5**

## **Integrated assessment of Spanish heritage learners in a high school medical interpreter college/career program**

### Kendra V. Dickinson<sup>a</sup> & Glenn Martinez<sup>b</sup>

<sup>a</sup>Rutgers University <sup>b</sup>University of Texas at San Antonio

Medical interpreters play an important role in bridging the language gap in health care and in reducing health disparities in minority language populations. The IM-PACT (Interpreters for the Medical Profession through Articulated Curriculum and Training) program is a medical interpreter for Spanish Heritage Language (HL) speakers program at a career academy in central Ohio. The goals of this program include meeting the demand for language access in central Ohio while at the same time valuing HL learners' experiences including language brokering (cf. Buriel et al. 1998), affirming their linguistic assets, and providing opportunities for them to develop competencies for future career and academic success. In this chapter, we report on the development, implementation, and evaluation of this program. We provide an integrated assessment approach that draws on qualitative and quantitative measures to highlight the role of program participation on students' college and career readiness. Using a variety of methods including interviews, quantitative measures of career decision, and academic data, we demonstrate that participation in the IMPACT program positively influences students' language proficiency, language attitudes, and career decision self-efficacy. In doing so, we provide a multifaceted perspective on HL learner achievement that is consistent with the goals of HL education (Beaudrie et al. 2014), that allows for integrated assessment of student development and career-readiness from a variety of different perspectives, and further demonstrates the importance of assessment models that consider the unique linguistic, social, and cultural assets of HL students.

Kendra V. Dickinson & Glenn Martinez. 2022. Integrated assessment of Spanish heritage learners in a high school medical interpreter college/career program. In Eva Rodríguez-González & Rosita L. Rivera (eds.), *Language assessment in multilingual settings: Innovative practices across formal and informal environments*, 105–126. Berlin: Language Science Press. DOI: 10.5281/zenodo.6762278

Kendra V. Dickinson & Glenn Martinez

### **1 Introduction**

Medical interpreters play a key role in bridging the language gap in health care and in reducing health disparities in minority language populations. Notwithstanding their crucial role in addressing language barriers, there is a national shortage of certified medical interpreters. A 2015 report from the Leonard D. Schaeffer Center for Health Policy and Economics at the University of Southern California found that there were only 738 certified medical interpreters in the state serving a population of 6.8 million Californians with limited proficiency in English (LEP, Gonzales 2015). The present shortage, furthermore, will be exacerbated in the future as the Bureau of Labor Statistics predicts a 20% increase in jobs for interpreters and translators by 2029 (Bureau of Labor Statistics 2020). The shortage of professional interpreters, as Dwyer (2001) explains, is a community problem that requires a community-oriented and strengths-based solution (Delgado-Romero et al. 2018). Career and technical education (CTE) programs represent a significant community resource for addressing the shortage of medical interpreters.

Current approaches to CTE differ significantly from earlier approaches commonly known as vocational education. As explained by Brand et al. (2013), CTE "is eliminating vocational education that consisted of low-level courses, job training and single electives and replacing it with academically rigorous, integrated, and sequenced programs of study that align with and lead to postsecondary education" (2013: 2). The shifting demographics of CTE participation, moreover, present an unparalleled opportunity for the inclusion of medical interpreting in CTE targeted at Spanish heritage language (HL) learners. Within less than a decade, Latinx participation in all CTE in the United States has nearly tripled from a mere 757,952 participants in 2009–2010 to a total of 2,085,517 participants in 2016–2017 according to data available in the Perkins Data Explorer (https:// perkins.ed.gov/pims/DataExplorer). Additionally, Latinx students accounted for only 9.9% of the total number of CTE participants in 2009–2010, but that figure rose to 25% by 2016-2017. These data suggest that Latinx students are gravitating towards CTE in greater numbers and that they are doing so at a pace that far exceeds other groups. CTE enrollments in health science cluster programs such as nursing, clinical lab sciences, and health information management, moreover, were the third most popular concentration options in 2016-2017. Between 2009– 2010 and 2016–2017, enrollments in health science cluster programs grew 44% and by 2016–2017 CTE concentrations in the health science cluster accounted for nearly 11% of all CTE enrollments according to data available in the Perkins Data Explorer. These trends suggest that CTE programs may constitute an *un-*

### 5 Integrated assessment of heritage learners in a medical interpreter program

*tapped pipeline* of medical interpreters to meet the health care needs of a growing LEP population and to simultaneously address the national shortage of certified medical interpreters.

In addition to the shifting demographics of CTE participation, programmatic changes also favor the inclusion of medical interpreting in CTE for HL learners. A significant opening for innovative curricula that span the high school/college/career pipeline has been created due to the following factors: (1) The reorganization of curriculum around clusters of occupations that share a similar knowledge and skill base, and (2) the formulation of career pathways that provide a coherent program of study blending high-level academics with technology applications and work-based learning (Castellano et al. 2003). The evolving frameworks of Perkins legislation including the integration of academic and vocational education (Perkins II), the flexibility of state and local agencies to develop CTE programs (Perkins III), and the strengthening of connections between secondary and postsecondary education (Perkins IV) provide additional openings for the incorporation of medical interpreting in CTE (Jocson 2018). Medical interpreting has been identified as a career specialty within the Support Services pathway of the Health Science Career Cluster (Advance CTE 2012). Even so, few CTE programs have established medical interpreter training within their health science clusters.

This chapter reports on the development, implementation, and evaluation of a novel medical interpreter CTE program for Spanish HL learners established at a career academy in central Ohio. In doing so, we demonstrate an integrated assessment approach that draws on qualitative and quantitative measures to highlight the program's impact on college and career readiness among Spanish HL learners. Much of the literature on HL assessment has focused on differentiating HL learners from second language (L2) learners and appropriately placing them in HL course sequences (Thompson 2015; Potowski et al. 2012; Vergara Wilson 2012; Fairclough 2006, Fairclough 2012; Beaudrie 2012). Other research has focused on formative (Carreira 2012) and summative assessment (Parra et al. 2018) in Spanish HL courses and programs. An integrated assessment approach differs from these previous approaches because it provides a multi-faceted perspective on HL learner achievement that is consistent with the goals of HL education. Beaudrie et al. (2014) propose the following seven HL education goals: language maintenance, acquisition or development of a prestige language variety, expansion of bilingual range, transfer of literacy skills, acquisition or development of academic skills, positive attitudes toward both the HL and various dialects of the language and its cultures, and acquisition or development of cultural awareness. The integrated assessment approach we adopt in this chapter seeks to provide a snapshot of HL

learner achievement by focusing on multiple metrics that address these goal areas including language proficiency, academic performance, career self-efficacy (an individual's belief that they can perform tasks and make decisions related to a career path: Taylor & Betz 1983), and ethnolinguistic identity. As described in more detail later in the chapter, we used a variety of instruments and metrics to piece together an overall profile of IMPACT participants. These included standardized tests, language proficiency assessments, self-efficacy instruments, and in-depth interviews with students. After a brief description of the IMPACT program and the setting where it was implemented, we will describe the specific methods used for the integrated assessment and present both the qualitative and quantitative findings.

### **2 The IMPACT Program**

The IMPACT (Interpreters for the Medical Profession through Articulated Curriculum and Training) program is a high school-university-industry partnership designed to meet the demand for language access in the central Ohio region through a college and career readiness program for HL learners of Spanish. It creates a high school level pathway in medical interpreting to complement other health science CTE programs in Pre- and Multi-Skilled Nursing, Dental Technologies, and Medical Data Management. Along the way, students earn 12 college credits in Advanced Spanish, develop college readiness skills through mentoring, and gain experience in medical interpreting. The program consists of four phases, illustrated in Figure 1.

Figure 1: Four phases of the IMPACT Program

*Phase 1* includes pre-program academic preparation and admission to the program. Students complete a two-year high school curriculum of Heritage Spanish during their freshman and sophomore years. Students who excel in these

### 5 Integrated assessment of heritage learners in a medical interpreter program

courses are identified by the Spanish teacher as potential candidates for the IM-PACT program. Potential candidates are recruited to the program through a series of coordinated activities during the second semester of their sophomore year. The recruitment activities begin with an after-school information session including Spanish teachers, IMPACT program faculty and staff, and current or former IMPACT students. The information session covers the goal of the program, its structure and requirements, and the application process. Students who indicate interest in the program after the initial information session are invited to an evening meeting with their parents. At this meeting, IMPACT faculty and staff and high school counselors and administrators discuss the need for medical interpreters and bilingual health professionals in the central Ohio region and describe how the program will help students fill this need. Parents are also informed of the benefits and obligations of the state-sponsored College Credit Plus program. After the meeting with parents, students participate in an after-school application workshop in which university admissions officers are on hand to assist in completing their college application. *Phase 2* consists of college level coursework. Upon admission to the university, students enroll in one advanced level Spanish course per semester for four semesters. The courses that students take are: Language, Culture and Communication in Latino Health (Spanish 2504); Advanced Spanish for Heritage Speakers (Spanish 3413); Translation and Interpreting in the Latino Community (Spanish 4369S); and Spanish in Health Care (Spanish 5201). *Phase 3* consists of career preparation and exposure. In this phase, students complete The Community Interpreter (TCI®) interpreter training, a widely recognized curriculum for entry level interpreter training, and complete a 50-hour internship consisting of patient assistance and interpreter shadowing at a local hospital. *Phase 4* consists of post-secondary education enrollment or entry into the profession. In this phase, students may enroll in college or sit for the National Board of Certification for Medical Intepreters (NBCMI) exam or the Core Certification Healthcare Interpreter (CCHI) national medical exam. Table 1 lists the activities included in the program and their alignment with the Knowledge and Skill Statements for the Support Services Pathway in the Health Science Career Cluster (States' Career Cluster Initiative 2008).

The program was implemented at the South-Western Career Academy (SWCA) which serves South-Western City School District in Franklin County in central Ohio. The Latinx population in Franklin County has grown 139% since 2000. This county has the largest percentage of Latinx residents in the state representing 5% of the total population according to the Pew Research Center. South-Western City School District serves the southwestern area of the City of Columbus and the Columbus suburbs of Grove City and Galloway. Latinx residents in the district



### Kendra V. Dickinson & Glenn Martinez

### 5 Integrated assessment of heritage learners in a medical interpreter program

make up between 8 and 20% of the population. Latinx presence in the district represents 16.3% of the total student population according to the Ohio Department of Education's (2017) CTE Report Card for South-Western City School District. SWCA offers CTE programs in 15 areas and, in 2018–2019, enrolled a total 4,708 students from Westland High School, Franklin Heights High School, and Central Crossing High School. Approximately 16.7% (796) of students at SWCA identify as Latinx. Only 9% of students at SWCA were classified as English Language Learners (ELL). Nearly half of the Latinx student population in the lower grades had exited the ELL program by the time they arrived at SWCA while maintaining strong Spanish language skills. SWCA offers four clinical concentrations in the health science cluster including Medical Data Management, Multi-Skilled Nursing, Dental Technologies, and Pre-Nursing. The total number of health science concentrations in these three programs is 174. A total of 58 students (33%) identify as Hispanic, Latina/o or Latinx.

The IMPACT program began in 2016 and has enrolled a total of 31 students to date. A total of 20 students have completed the program and graduated from high school. All students who completed the program identified as Hispanic/Latinx. The majority of students were of Mexican origin and three students had ancestry in Peru, El Salvador, and Ecuador. 90% of program completers (n = 18) graduated high school with an overall GPA of 3.0 or higher. Seventy percent of program completers (n = 11) have continued on to enroll in higher education and 35% (n = 7) have received a full-ride scholarship to attend a four-year college or university. Twenty five percent of program completers (n = 5) have gone on to earn credentials or employment in the field of medical interpreting.

### **3 Findings**

Our integrated assessment sought to gain an in-depth perspective of student achievement in and beyond the IMPACT program. In order to do so, we collected and analyzed data on a set of measures that align with the goals of HL education. These measures consisted of Spanish language proficiency assessed with the Parrot Language Proficiency Test, English language achievement assessed with the Ohio ELA test, academic achievement assessed with the College and Career Readiness (ACT) exam and high school grade point average (GPA), career decision making assessed with the Career Decision Self-Efficacy Scale, and ethnolinguistic identity and attitudes towards the HL assessed with in-depth individual interviews. In sum, our goal was to profile the IMPACT graduate in terms of language proficiency, language attitudes, and career decision making and at

the same time compare the IMPACT graduate with other HL students with respect to academic achievement. Our integrated assessment approach thus sought to answer two research questions:


### **4 What is the profile of the IMPACT graduates?**

Previous research has shown that different factors can influence student academic outcomes and career and academic self-efficacy. Some of these influences include career pathway programs and career counseling (Stipanovic et al. 2017), language brokering experiences (Buriel et al. 1998), student perception of barriers (see Luzzo & McWhirter 2001; *inter alia*), student support networks (e.g. teachers, friends, family; cf. Berbery & O'Brien 2018; Carpi et al. 2017) and student linguistic and ethnic identity (Mejia-Smith & Gushue 2017; Ojeda et al. 2012). Traditional methods of assessment for HL learners often focus on tests, structured classroom activities, and analyses of HL production. Our integrated assessment approach sought to combine objective assessments of language proficiency and career decision making together with students' reflections on their own academic experiences, experiences in the IMPACT program, and their future careers. With this approach, we aimed to provide important insights into student learning and development outcomes.

### **4.1 Language proficiency**

Thirteen IMPACT graduates took the Parrot Language Test (PLT). The PLT is a remote language testing system that measures functional abilities in the language based on the Interagency Language Roundtable (ILR) proficiency scale. The PLT uses multi-stimulus prompts including audio prompts, on-screen text and video accompaniment to generate evaluated speech. Results are based on agreement of three blind ratings that have been shown to achieve 90% reliability (Parrot 2019). As a workplace language testing system, the PLT isolates three levels of workplace proficiency as follows:


77% percent (n = 10) of IMPACT graduates that took the PLT were certified at the ILR 3 level while 23% (n = 3) were certified at the ILR 2+ level.

### **4.2 Language attitudes and ethnolinguistic identity**

In order to explore language attitudes and ethnolinguistic identity among IM-PACT graduates, we created a series of open-ended questions aimed at gaining insight into students' future career and educational goals, career outlook, language use and identity, mentorship and role models, community, perceived barriers to success, and overall attitude towards the IMPACT program. We then set up conversational-style one-on-one interviews via Zoom between individual IM-PACT students and their former Heritage Spanish high school teacher, organized around the questions that we created. Interviews lasted between 35–90 minutes, and students received a \$15 gift card as compensation for their participation. A total of 11 IMPACT graduates were interviewed.

The interview data pointed to substantive gains among HL students in terms of positive attitudes towards Spanish and cultural awareness of Latinx communities. Students interviewed described how participation in the program contributed to greater confidence in their language abilities and to a renewed sense of pride in their cultural heritage. In terms of language abilities, students commented that they no longer felt ashamed of their Spanish after participating in the IMPACT program. One student commented:

So, I would say like, in the interpreting thing-y, I would say like, I realized like, you know, I, I shouldn't be ashamed of speaking two languages. There's some people out there, that are just like, I don't know, they're just ashamed to speak the language. Cause there's like some Latino who like, they speak English and Spanish, but they're like embarrassed to speak Spanish, you know? I, I really don't care what people think.

In terms of cultural heritage, on the other hand, students commented that they came out of the program with a greater appreciation for the variety of Latinx cultures in the U.S. For some students, this new appreciation led to a more positive perception of their own cultural heritage. One student put it this way:

Uh, I mean, uh, not gonna lie. Back then I was a little scared to say I was Salvadoran. 'Cause not a lot of people are Salvadoran, and there's like a lot of Mexicans. Now I'm not really scared to be like "Oh, yeah, I'm from El Salvador." It's a little country, but it's there.

Shifting attitudes about Spanish were not only reflected in students' perception of themselves but also in their perception of the utility and value of the language in their future. One student commented on the economic advantage that he had because of his bilingualism:

We both have the same skills, but then here's one difference: He speaks just English and I speak two. So, I have higher chance of getting hired than he would. Just like, small things like that, you know? That's reality. Um, it, it makes you look smarter, in some way, you know, just because, hey he speaks two languages, you know, he only speaks one. And it's, imagine if someday I would speak three languages, and it's like … Wow. He's like super smart.

Another student reflected on the cultural advantage of her bilingualism:

I feel like because I did this, I want to communicate more with people of all different types of, like all different types of Latinos, any part of like Latin America. I'd like to com- I would like to communicate with all of them and learn more about their traditions, their cultures, their music, their food, everything.

Even while students commented on the positive benefits and advantages of bilingualism for themselves, they were equally emphatic about how their bilingualism would allow them to help others. One student commented: "I really like using Spanish, especially when it comes to helping others." Another student expressed the same sentiment in greater detail:

And a bunch of like, Latinx people move to the United States because they want a better future for their children. Um, like the DREAM Act and all of that stuff. And, and there's this kinda p- pressure not to fail. But, at the same time, it's, it's, it's taking that dream and transforming it into your own.

### 5 Integrated assessment of heritage learners in a medical interpreter program

And how, like, my identity makes me want to do great things, because of my parents, you know? They sacrificed so much. I wanna do great things. But also, it, it allows me to go back and understand that the things that happened to me, I can shape them differently for another person, or I can help um, create uh, a social community, or uh, create this kind of community that helps our youth, and helps those that are now living through the problems today.

Our interview data showed that IMPACT graduates emerged from the program with positive attitudes about their Spanish and greater awareness of the cultural variety of Latinx communities. At the same time, they emerged with a clear view of the advantages of being bilingual and with a commitment to using their bilingualism to help others.

### **4.3 Career decision making**

Part of our integrated assessment also included a measure of career decision selfefficacy. Career decision self-efficacy has been shown to be related not only to career development among Latinx high school students, with respect to both vocational identity and career exploration activities, but also to student perception of barriers (Gushue et al. 2006). To measure levels of career decision self-efficacy among students in the IMPACT program, we utilized the Career Decision Self-Efficacy Scale-Short Form (CDSE-SF, Betz & Taylor 2012). In this instrument, students rated their degree of confidence in successfully completing 25 careerrelated tasks on a 5-point Likert Scale, where higher scores indicate higher levels of career decision-making self-efficacy. We administered the online version of the CDSE-SF to a total of 17 IMPACT graduates who elected to participate. Students were compensated with \$10 gift cards for their participation.

Items on the CDSE-SF are grouped into five principal areas: Goal selection, occupational information, planning, problem solving, and self-appraisal. Table 2 below shows the average scores in these areas among this group of IMPACT students.

This table shows that students scored, on average, between 3.76–4.02 in all areas targeted on the CDSE-SF. Importantly, students scored, on average, above 3.5 in all areas, which is considered to indicate "Good confidence: Comfortable with this skill set" (Betz & Taylor 2012). However, there is slight variation among individual students, as shown in Table 3.

Even though the majority of students scored within the highest bracket of scores (3.5–5.0) in all cases, there are cases in which students scored in the lower


Table 2: Mean CDSE-SF Scores by area


### Table 3: Distribution of CDSE-SF scores of HL students

### 5 Integrated assessment of heritage learners in a medical interpreter program

range. This breakdown allows us to identify areas in which students need additional support. For example, just one student scored within the 1.0–2.5 range in one area, occupational information. Using the CDSE-SF as an assessment metric for HL students would allow us to provide targeted intervention regarding career resources to this student in particular. Similarly, this method of assessment identifies the 11.8%–35.3% of students scored within the 2.5–3.5 range in all breakdown areas, allowing the IMPACT program to identify groups of students that need additional support and to provide interventions in those areas. Still, overall results show that students in the IMPACT program show good confidence levels in the areas targeted by the CDSE-SF, demonstrating that they are well-positioned for their future careers with regard to career decision-making self-efficacy.

### **5 How do IMPACT graduates compare to non-IMPACT Latinx graduates?**

Another area of inquiry is the relationship between IMPACT program participation and student academic outcomes. In this analysis, we consider two conventional measures of student academic achievement: GPA and ACT exam scores. In our analysis, we find that Latinx students who participated in the IMPACT program were significantly more likely to have higher GPAs and ACT scores than Latinx students who did not participate in the program.

Data are drawn from students who took high school level Heritage Spanish I and II during the 2017–2018, 2018–2019, and 2019–2020 academic years. All students who were subsequently enrolled in the IMPACT program first took this course, but not all students who take the course ultimately enroll in the IMPACT program. Therefore, we are able to analyze the GPA and ACT scores of students who took the Heritage Spanish courses and did not enroll in IMPACT and students who did.

Along with student GPA and ACT scores, we also considered a number of additional factors as potential predictor variables. We considered students' scores on Ohio Department of Education (2019b) English Language Arts (ELA) tests 1 and 2 , taken in their freshman and sophomore years in high school, before enrollment in the IMPACT program, and student gender, school year, and IMPACT program participation as potential predictor variables.

In order to evaluate the potential relationships between IMPACT program participation and these metrics, data were examined and analyzed in R (R Core Team 2017) using the lme4 package (Bates et al. 2015). First, we used a step-up method

### Kendra V. Dickinson & Glenn Martinez

to build series of linear regression models of the data by sequentially added predictor, and then compared them using the ANOVA function in R to determine the best-fit models for our data.

### **5.1 Student GPA**

Our first model compared the GPA scores of students who took the Heritage Language course and did enroll in IMPACT (n = 19) and those who took the course but did not enroll in IMPACT (n = 32). This model included student GPA as the dependent variable, and IMPACT program participation, school year, English Language Arts assessment 1 and 2, and student gender as potential predictor variables. The output of the best-fit model is shown in Table 4 below.

Table 4: Best-fit model of student GPA


The output shows the GPA of the students in the IMPACT group as compared to the GPA of the students in the reference level, the non-IMPACT group. None of the other factors considered were found to be statistically significant predictors of student GPA. The positive estimate for the IMPACT group (1.3882) shows that IMPACT students tended to have higher GPAs (mean = 3.7, standard deviation (sd) = 0.61) than non-IMPACT students (mean = 2.3, sd = 0.52). Furthermore, this difference is statistically significant, where IMPACT program participation significantly predicts higher GPA for those who participated in the program than for those that did not. This difference is visualized in the boxplot Figure 2 below.

The potential influence of the IMPACT program on student GPA has far-reaching implications. For example, in their study of Latinx high school students, Berbery & O'Brien (2018) found that grade point average was the most influential factor in student college-going self-efficacy and educational goals. The IMPACT program motivates students to achieve higher GPAs in high school.

### 5 Integrated assessment of heritage learners in a medical interpreter program

Figure 2: Boxplots of IMPACT vs. Non-IMPACT student GPA

### **5.2 ACT Scores**

Another conventional metric of academic achievement that we considered in the present analysis is student ACT score. For this analysis, we had access to ACT scores from 26 non-IMPACT graduates and 19 IMPACT graduates. For each student, the ACT score considered here is taken from a school administered, required test, taken by the student during their junior year. This model included student ACT as the outcome variable, and IMPACT program participation, school year, ELA assessments 1 and 2, and student gender as potential predictor variables. The output of the best-fit model is shown in Table 5 below.

This output shows the statistical relationship between student ACT scores and IMPACT program participation, student ELA 2 assessment level, and school year. First, this output shows that IMPACT students were statistically significantly more likely to score higher on their ACT exam (mean = 18.47, sd = 2.94), than non-IMPACT students (mean = 14.65, sd = 2.82). Another significant predictor of student ACT scores shown above is the ELA 2 test score. This test is administered to students in their sophomore year of high school, as part of requirements established by the Ohio Department of Education, though this assessment has since been eliminated for future students (Ohio Department of Education 2019a). The ELA tests have 5 potential levels of achievement, which are, in order of lowest to highest: Limited, Basic, Proficient, Accelerated, and Advanced. Students in

### Kendra V. Dickinson & Glenn Martinez


Table 5: Best fit model of student ACT score

the current sample achieved only the first four levels. The model output above shows that students who achieved "Limited" and "Basic" proficiency scores on their ELA 2 exam also scored significantly lower on their ACT exam than did students who received a rank of "Accelerated." Furthermore, students who received a "Proficient" level on their ELA 2 exam did not receive ACT scores significantly different from those who achieved a rank of "Accelerated." Releveling of the factors in the model revealed that the ACT scores received by students who scored "Limited" and "Basic" on their ELA 2 exam were not significantly different from each other. These differences are visualized in Figure 3 below.

Finally, there is also a main effect of school year in these data. ACT scores for the group of students that we have data for were significantly lower in the 2018–2019 school year than they were for the 2017–2018 and 2019–2020 school years. There may be real-world explanations for this observation, such as the conditions under which the exams were administered, or this may simply be an outlying year. Overall, these results illustrate that IMPACT program participation is the strongest predictor of ACT scores for this group of HL students, and that ELA 2 tests scores and school year also play a role in these outcomes.

### **5.3 IMPACT student changes in ACT scores**

Because students take the school district-administered ACT exam during their junior year, many students will take the exam again. Therefore, we were able examine changes in ACT score for a subset of IMPACT students for whom the

### 5 Integrated assessment of heritage learners in a medical interpreter program

Figure 3: Boxplot of student ELA 2 scores

data was available (n = 16). Their first scores are from the ACT exam taken during their first year of participation in the IMPACT program, while their second scores are from their second year in the program.

In order to conduct paired t-tests on the data, we first needed to ensure normal distribution, due to the small sample size, using a Shapiro-Wilk test of normality (Shapiro & Wilk 1965). Outputs of this measure show that neither students' first ACT score (p = 0.2659) nor students' last ACT scores (p = 0.3635) were significantly different from normal distributions. This allowed us to more confidently conduct paired t-tests on the data. We then conducted a paired t-test to compare the means of two related samples, the students' first and last ACT scores. The output of this test shows a significant difference between the two (t = −2.6379, df = 15, p-value = 0.01864), where students achieved significantly higher scores on their last ACT exam than on their first. This difference is visualized in Figure 4 below.

Furthermore, it is also of interest to examine how the ACT scores of individual students differ from their first to their last ACT exam. Figure 5 below shows individual plots by student tracking the change between their first and their last ACT scores. From this image, we can see that the majority of individual students (68.75%, n = 11) scored better on their second ACT exam, whereas only 25% (n = 4) received the same score on both exams, and 6.25% (n = 1) received a lower score on their second exam.

Figure 4: Boxplot of IMPACT student first and last ACT scores

Figure 5: IMPACT student first and last ACT score by student

### **6 Conclusion**

In this chapter, we have demonstrated an integrated assessment approach that draws on qualitative and quantitative measures to highlight the program's impact on college and career readiness among Spanish HL learners. As opposed to focusing on differentiating HL learners from L2 learners, we have provided a multi-faceted perspective on HL learner achievement that is not only consistent with the goals of HL education, but that also allows for an integrated assessment of student development and career-readiness from a variety of different perspectives.

First, we have shown that language proficiency, language attitudes, and career decision self-efficacy were affected by participation in the IMPACT program. IM-PACT students demonstrated high performance on a workplace language proficiency assessment. At the same time, students indicated more positive attitudes to Spanish and were able to make explicit connections between Spanish language proficiency and career opportunities. Second, we have shown that measures of career-decision self-efficacy, in this case the CDSE-SF, can be useful measures of career-readiness that can also allow the program to identify areas of student need and implement appropriate interventions. Third, we have shown the participation in a program like IMPACT has the potential to affect student outcomes in academic areas. For example, graduates of the IMPACT program had statistically significantly higher GPAs and ACT scores than HL students who did not participate in the program. We also saw the ACT scores generally increased for students after enrollment in the IMPACT program.

The student outcomes reported in this study reflect an innovative approach to heritage language teaching that connects language and culture to an obvious and immediate community need. By exposing heritage learners to the medical interpreting profession in a CTE context early in their schooling careers, students are given an opportunity to tap into cultural and linguistic funds of knowledge in a way that is often not available in mainstream CTE or Spanish language classes. The integrated assessment approach described in this chapter demonstrates the multiple effects that participation in this program had on academic, career and linguistic outcomes.

Our integrated assessment approach provides insights into the synergies between linguistic, academic, and career development of HL students and shows how this combined achievement interacts with affective dimensions of HL learning including the development of positive attitudes about Spanish and confidence in the advantages that the HL will provide in the students' future career. We believe that innovative college and career oriented programming together with con-

tinuous integrated assessment can more effectively connect heritage language instruction with student success and thus contribute to a robust evidence base demonstrating the immense value of HL learning for Latinx students.

### **References**


### 5 Integrated assessment of heritage learners in a medical interpreter program


gov/getattachment/Topics/Testing/Sections/Resources/Elimination-of-ELA. pdf.aspx?lang=en-US (15 October, 2020).


## **Chapter 6**

## **Assessment in community-based heritage language programs: The case of Brazilian Portuguese**

### Gláucia V. Silva

University of Massachusetts-Dartmouth

In heritage language (HL) education, testing for administrative motives (i.e., student placement) has received special attention (Fairclough 2012b), and with good reason. Given the heterogeneity that characterizes HL learners, it is important to be able to place them adequately. However, it is also important to be able to determine whether learning goals are being met. According to Carreira (2012a), it is essential that educators utilize formative assessment in HL classes, which would allow them to address issues of learner diversity. Most of the literature dealing with HL assessment, however, is based on university-level education. We know little about how community-based HL schools in the United States assess learner progress and determine student readiness, and even less, if anything, about the assessment of linguistic and cultural skills in less commonly taught HLs, such as Portuguese. This chapter aims to shed some light on issues of placement and of assessment of learning in community-based HL schools by presenting data from a survey distributed to Brazilian Portuguese language teachers and school administrators in the U.S. Results indicate that there is an array of behaviors in relation both to administrative and to instructional assessment in these schools, which range from grouping only by age to using tests to place learners and assess their progress. Based on the available literature and on the data analyzed, the chapter also presents suggestions regarding assessment in Brazilian Portuguese community-based HL programs and possibly others.

Gláucia V. Silva. 2022. Assessment in community-based heritage language programs: The case of Brazilian Portuguese. In Eva Rodríguez-González & Rosita L. Rivera (eds.), *Language assessment in multilingual settings: Innovative practices across formal and informal environments*, 127–151. Berlin: Language Science Press. DOI: 10.5281/zenodo. 6762280

Gláucia V. Silva

### **1 Introduction**

For the past few decades, researchers have emphasized that heritage language (HL) learners present specific needs that differ from those of foreign language (FL) and of monolingual learners. According to Valdés (1981), HL teaching should provide opportunities for learners to develop their oral proficiency and their listening skills, which are normally at a higher level than that presented by FL learners, besides their reading and writing abilities. The expansion of learners' bilingual range, according to Valdés (1997), should be one of the goals of a heritage language class. She argues that this expansion may be difficult to attain in courses designed for FL learners.

Besides exhibiting different pedagogical needs from those of FL learners, HL learners also diverge from their FL counterparts in relation to the spectrum of linguistic abilities. While learners in a beginning FL class generally (though by no means always) start out knowing nothing or very little of the language, HL learners display a wide range of abilities (e.g., Wang & Green 2001; Beaudrie et al. 2014). This spectrum of broad linguistic skills is captured in the definition of a heritage language learner proposed by Valdés (2001: 38), which refers to someone who speaks or at least understands the HL and is, therefore, bilingual to a certain extent. This range of abilities may be related to the special attention received by diagnostic assessment in HL education (Fairclough 2012b). Given the variation in linguistic abilities exhibited by HL learners, placing students in the adequate courses is essential.

Tests have ideological force, as pointed out by Leeman (2012: 54), and traditional language exams (i.e., those that emphasize grammar and spelling, for example) "devalue or erase the conversational, pragmatic, and cultural arenas where many SHL [Spanish as a heritage language] students excel." Recognizing that the needs of HL learners merit revisiting the way language testing was often carried out, researchers and practitioners have proposed exams that include ways of assessing the abilities that HL learners already possess and what they need to develop. However, much of the discussion revolves around university-level courses. We know little at this point about what happens in relation to assessment in HL community-based schools, especially in the case of languages that are less commonly discussed in the literature.

The Coalition of Community-Based Heritage Language Schools (2018) explains that, typically, HL community-based schools in the United States are nonprofit organizations founded and operated by members of the immigrant or heritage language community. The Coalition adds that the goal of these schools is to maintain and teach the language and culture of the immigrants' heritage (but

### 6 Assessment in community-based heritage language programs

note that such schools may also teach and maintain indigenous languages/cultures), and that they may offer classes for learners from Pre-K to Grade 12. HL community-based schools often operate in rented spaces on weekends or after school during the week. Even if these spaces are located on public or private school premises, community-based schools are not connected to school systems, nor are they subject to the regulations of the U.S. education system. Regarding assessment, the Coalition states that community-based schools may choose to administer U.S. language tests or tests used in the home country when appropriate assessments are available.

Lu (2020) reports that there are 29 Portuguese language community-based schools in the U.S., a number based on a survey made available by the Coalition of Community-Based Heritage Language Schools (https://www.surveymonkey. com/r/HLProgram). However, many Portuguese language community school leaders have not completed the survey. The 29 programs reported by Lu include both European and Brazilian Portuguese community-based schools; this chapter reports on assessment practices in Brazilian schools only. A discussion of the differences between European and Brazilian Portuguese is outside the scope of this text. For our purposes, suffice it to say that, although there are schools that serve mainly the Brazilian community and others that serve mainly the Portuguese community, sometimes students of Brazilian heritage attend a community-based school of Portuguese heritage, or vice-versa.

One example of Brazilian Portuguese community-based school in the United States is ABRACE (*Associação Brasileira de Cultura e Educação*), which was founded by three immigrant Brazilian mothers (Center for Applied Linguistcs 2013, ABRACE 2020). ABRACE's mission is "to preserve and promote Brazilian education and cultural and social integration in the Brazilian community with the aim of strengthening its identity within the United States" (Center for Applied Linguistcs 2013). Funding for ABRACE comes from student tuition, occasional sponsorship by Brazilian companies that operate in the U.S., individual donors, and occasional support from the Brazilian government. Many other Brazilian Portuguese programs, however, are funded solely through student tuition. In relation to assessment, the information for ABRACE mentions that there are no standardized tests available for children and youth (Center for Applied Linguistcs 2013).

This chapter illustrates what may happen in HL education of young learners by discussing assessment practices in several Brazilian Portuguese communitybased schools. Starting from what is already in use, the chapter also outlines suggestions of assessment tools that can be utilized by community-based Brazilian Portuguese HL schools and possibly others.

### Gláucia V. Silva

After this brief introduction, this chapter is divided as follows. §2 presents some of the previous research on HL assessment and is followed by a discussion of community-based schools (focusing on Brazilian Portuguese) in §3. §4 introduces the study, including methodology and participants, while §5 presents the research findings. A discussion of those results, as well as suggestions regarding assessment, are presented in §6. Final remarks are offered in §7.

### **2 Assessment in heritage language education**

The HL education field has been growing steadily since at least the early 1980s, when Valdés et al. (1981) published their seminal edited volume on teaching Spanish to heritage leaners (at that point, referred to simply as "bilinguals", which they are). In that volume, the chapter by Janet Ziegler (1981) discusses placement examinations, calling attention to which skills, according to her, should be tested when placing heritage speakers of Spanish into Spanish language courses, including issues related to morphology and syntax.

The importance of assessment for placement purposes is evident in the HL education field, as pointed out by Fairclough (2012b) and Carreira (2012a), due to the great diversity found among HL learners. MacGregor-Mendoza (2012) calls for placement tests that are informed by current research and that tap into learners' oral, aural, and pragmatic abilities, and include a background survey (which, in the case of children, may be completed by parents or caretakers) as well as face-to-face interviews. Fairclough (2012a: 126) suggests that HL placement tests should measure three general areas: receptive skills (such as general vocabulary), productive skills (focusing on linguistic gaps, dialectal forms and language transfer), and creative skills (speaking and writing, if appropriate, that reflect a range of functions and contexts). Carreira (2012b) argues that the data gathered from placement exams should be used to inform syllabus design on a regular basis (a point also defended by Ilieva & Clark-Gareca 2016).

In a review article focusing on assessment of HL learning at the university level, Son (2017) shows that discrete-item tests are the most common type of placement exam for that level. However, these exams often utilize a combination of methods to better assess students' abilities. Son (2017) espouses (as do Beaudrie & Ducar 2012) the notion that placement exams must address the needs of specific programs and students. The idea that each program needs its own exam may account for why HL educators may not be able to use or adapt placement tools made available by the National Heritage Language Resource Center (https://nhlrc.ucla.edu), as Carreira (2014) notes.

### 6 Assessment in community-based heritage language programs

In a discussion about assessing the language of young learners, Bailey (2017) calls attention to complexities involving bilingual first language acquisition, including the fact that children who acquire more than one language simultaneously may become literate in only one of their languages. For diagnostic purposes, Bailey suggests that processing demands may be lessened with verbal scaffolding that would help elicit responses from young test takers, a strategy that would generate diagnostic information.

Beyond placement tests, assessment is often categorized as either summative or formative. Summative assessment is an exam that "evaluates learning after instruction for purposes of assigning a grade or determining the efficacy of particular programs or interventions" (Carreira 2012a: 100). Carreira (2012a,b) asserts that formative assessment, conceptualized as assessment for learning (as opposed to assessment of learning), is ideal for HL education. She ties formative types of assessment with differentiated teaching, which, she argues, is an instructional approach that meets the pedagogical needs of HL learners, given the diversity found among these students (Carreira 2012b). Among the activities that lend themselves to formative assessment, Carreira (2012a) lists exit cards, journals, portfolios, surveys, oral interviews, and presentations. She maintains that these types of activities provide information about each learner, making them ideal for differentiated instruction: instructors can assess differences among learners' linguistic abilities as well as attitudes and goals, and then attend to them. Beaudrie (2016: 152) goes a step further and states that "[d]ifferentiated assessment complements differentiated instruction, seeking to provide all students with multiple opportunities to demonstrate their learning and progress."

Bailey (2017) maintains that assessment *for* learning is especially relevant in the case of young learners, since they are still acquiring the language. She adds that formative assessment "can capture a broad array of relevant language information for teachers that is closely tied to the young learners' instructional needs" (Bailey 2017: 329). A central focus of formative assessment, Bailey states, is teacher feedback to students; students may also self-assess their language learning.

Formative assessment can make use of performance-based tasks, as argued by Ilieva & Clark-Gareca (2016), who highlight the need to take into consideration the abilities of HL learners in their totality. Following Brown & Abeywickrama (2010), Ilieva and Clark-Gareca contend that simulations of real-world tasks lend themselves to testing of linguistic abilities in more authentic ways and that, through low-stakes performance tasks, "teachers can create an excellent learning environment that boosts student motivation and positive attitude toward learning the languages" (Ilieva & Clark-Gareca 2016: 227). The researchers recommend

### Gláucia V. Silva

a model of assessment that reveals nuances of learners' proficiency by incorporating multiple modality assessment strategies consistent with four principles: (1) centrality of authentic contexts; (2) multiplicity of measures; (3) diversity of feedback; and (4) reliance on research (Ilieva & Clark-Gareca 2016: 229–30). They maintain that this model offers valuable information to teachers on what kind of instruction, topics, and structures need to be practiced in the HL classroom.

Bailey (2017) highlights that the content of the assessment tasks must be relevant to young learners both in terms of cognitive demands and of cultural specificity. According to Bailey, the younger the learner, the higher the need for contextualization in assessment: items in a test need to be "topically appropriate for the target age of the test taker, and the ability to answer the items should not require knowledge of information not already provided in the tasks or test items" (Bailey 2017: 332). For the youngest learners, Bailey mentions that toys may be incorporated in questions and in response formats, since young children are more successful in production and comprehension tasks if objects rather than pictures are used. Still, Bailey recognizes that choosing age-appropriate content for tests for young learners is complex because language development is concurrent with developments in other areas.

Elder (2005) considers the role of testing in HL education in selected Australian schools and discusses dilemmas faced by evaluators in implementing testing programs and interpreting test results. The programs discussed by Elder included two schools for primary school-age children and two for secondary school-age learners. As mentioned above, HL community-based schools may serve learners of all the ages included in Elder's (2005) study.

Formal testing may receive more attention in educational systems such as those included in Elder's (2005) study than in HL community-based schools. However, both types of schools may face similar challenges in relation to assessment. Some of the issues mentioned by Elder are: lack of expertise (teachers may not be experts in language assessment); limited opportunities for test piloting; appropriateness of level for each learner; difficulty devising instruments that reflect a program's curriculum at a specific point and also serve as indicators of language achievement over a more extended period. Given these (and other) challenges, Elder outlines recommendations for effective use of heritage language tests in Australian schools. Elder highlights the importance of systematically documenting relevant aspects of HL learners' home language background, including language exposure, a recommendation that may be useful for HL community-based schools as well. Elder argues that such sociolinguistic profiling is essential in the beginning and throughout the program, and should be taken into account in the interpretation of test results.

### 6 Assessment in community-based heritage language programs

This section has reviewed some of the relevant literature regarding general assessment practices in heritage languages. The next section discusses communitybased heritage language programs, which strive to help children maintain their heritage language and culture, despite the many challenges faced, including the lack of adequate assessment instruments.

### **3 Assessment in community-based heritage language programs**

Research on the assessment of HL children tends to revolve around bilingual programs (e.g., Lucero 2018) or how to assess their progress in mainstream education (e.g., Gonzalez 2012). Practices in community-based schools have received less attention, even if, around the world, these programs support efforts to maintain and develop the linguistic and cultural skills of HL learners. Historically, these schools have been established and supported by groups interested in the maintenance and development of their languages and cultures (Bradunas & Topping 1988), though they may also be sponsored by both the community and a local public school or community college (Compton 2001). As Douglas (2005) notes, in the North American context most pre-college HL instruction is provided by these schools. Establishing and maintaining such schools involves many challenges, as outlined, for example, in Compton (2001) and Liu et al. (2011). Calling attention to the wide range of linguistic skills among HL learners, Compton (2001: 155) maintains that "new approaches to placement, testing, teaching and learning for heritage language students are crucial." At that time, Compton states, instruments for assessing HL learner skills were still in developmental stages.

The diversity of students' language skills and backgrounds is, according to Liu et al. (2011), one of the challenges faced by community-based HL programs. Although Liu et al. mention that educators "would like information about placing students when their proficiency levels and backgrounds differ" (Liu et al. 2011: 5), the suggestions offered to address this particular challenge focus on instructional approaches and materials, not on assessing learner level for placement reasons. Assessment of learning outcomes is not included in other challenges related to instruction either (such as teaching materials and instructional time), possibly because other issues are considered more pressing for those schools. However, community-based HL programs may not be seen as schools by teachers and administrators. In an analysis of the curriculum of a Brazilian Portuguese HL program, Boruchowski (2014: 83) explains that the limited hours of contact with the children and the fact that teachers do not "grade or use measurement tests to

### Gláucia V. Silva

evaluate their students" leads teachers and administrators not to classify their own program as a school. Thus, assessment (or lack thereof) may be directly related with how HL programs are perceived by those responsible for instructional decisions.

In an article that proposes a theoretical framework for curriculum development for Japanese HL schools, Douglas (2005: 71) highlights that assessment methods must reflect the principles that were used as a basis for the curriculum. She calls attention to the elements of ideal assessment listed by Gutierrez & Slavin (1992), the principles of which can serve as a framework for developing assessment tools for young HL learners. The first element in the list seems especially applicable to HL learners, given the range of abilities found among them: "Children are evaluated in terms of their own achievement and potential, not by comparison to group norms. Expectations differ for different children" (Douglas 2005: 71). The list also contains elements that relate to formative assessment, which, as Douglas (2008) and Carreira (2012a,b) maintain, is individualized in nature and, thus, necessary to address the needs of HL learners. Douglas (2005: 72) argues that assessment tools for young HL learners should address academic language as well as basic communication skills, given the varied HL environment to which children are exposed.

Douglas (2008: 256) specifies that "in order to assess oral language development, assessment is conducted whenever possible while students are engaging in authentic meaning making activities." Assessment of oral skills can be recorded in different ways, such as checklists, teacher observations, reports, and student interviews, among others. Douglas (2008) also discusses assessment of reading and writing abilities, providing examples of skills and categories that can be assessed. Results may reveal a mismatch between learners' needs and the curriculum objectives. When this happens, Douglas (2008: 259) recommends that instructional goals be adjusted, reestablished or repeated.

The evaluation of curriculum objectives based on assessment results is also proposed by Boruchowski (2015), who, following Wiggins & McTighe (2005), maintains that assessment should be used to assess whether instructional activities did in fact help young HL learners reach learning goals. The importance of establishing curriculum goals for young HL learners is highlighted by Silva & Boruchowski (2016), who recommend that educators examine learners' history of HL use and schooling in order to set specific objectives. Like Douglas (2005, 2008), Silva & Boruchowski (2016) also assert that educators use formative assessment for young HL learners, including checks for understanding as well as performance tasks and projects.

### 6 Assessment in community-based heritage language programs

Though not exhaustive, this literature review provides a base for discussing assessment in Brazilian Portuguese HL community-based programs. The next section describes the present study and is followed by a presentation of the answers elicited by the survey used.

### **4 The present study**

Heritage language community-based schools play an important role in preserving linguistic and cultural ties (Kondo-Brown 2010) and, thus, enriching any society. However, we do not have much information about how these schools assess learner progress and determine student readiness, and even less, if any, about the assessment of linguistic and cultural skills in less commonly taught HLs, such as Brazilian Portuguese. This study aims to shed some light on assessment practices in these HL schools and was guided by the following questions:


In order to answer these research questions, approximately 80 educators connected to HL schools that teach Brazilian Portuguese in the United States (specifically, in Massachusetts and in Florida) were invited, via email, to participate in an anonymous online survey. Many (perhaps most) of these schools were created by community members who may have identified a desire by Brazilian immigrants for their children to maintain and develop their language and culture. Blizzard & Batalova (2019) show that, according to American Community Survey data, approximately 450,000 Brazilian immigrants lived in the U.S. in 2017. The authors add that about 32% of those immigrants resided in Florida and Massachusetts. These patterns of immigration may account for the creation of several community-based schools in these two states. Although I do not have specific information about the schools these educators were connected with, in my experience most Brazilian Portuguese HL community-based schools are funded with student tuition, though some may establish partnerships with Brazilian companies and/or obtain limited funding from the Brazilian government, as mentioned earlier in the case of ABRACE.

### Gláucia V. Silva

About two weeks after the first email, another email invitation to take part in the anonymous online survey was sent to the same list. The survey contained nine questions (see Appendix A), four of which required an answer, while the other five were dependent on other questions, that is, five questions would be answered depending on the answer provided in the required questions. Six questions were open-ended, which leads to a mostly qualitative analysis of the results. Questions addressed placement, assessment during the academic year, and summative assessment. A total of 19 respondents completed the survey, but answers by one participant were excluded because it was evident that s/he does not work at a community-based HL school, but rather at a regular school (possibly in a bilingual program). Answers were collected within the online survey tool used to design and distribute the survey. The results are presented in the next section.

### **5 Survey results**

This section presents the answers to the online survey provided by 18 participants. The first question (Q1) in the survey addressed the period before classes started, asking whether anyone at the school spoke with the parents or guardians of new students. All of the respondents answered that there were in fact people who spoke with parents prior to the child joining the program and proceeded to answer the second question (Q2), which depended on an affirmative answer for the first and focused on the goal of the conversation. Since Q2 was open-ended, a few participants mentioned more than one objective for the conversation with the parents of new students (and, in the case of one respondent, parents of returning students as well). The answers revealed some common themes, as well as topics that were unique among the participants. The list below summarizes the reasons for the conversation with the parents of new students:


The third question in the survey (Q3) was one of the multiple-choice questions. It sought to elicit how new students were placed (and answer the first research question guiding this study). Possible answers were: (a) according to age only; (b) according to the result of an evaluation and/or interview; (c) according to age and the result of an evaluation and/or interview; and (d) other. No participant elected "other". Most respondents (n = 10) chose (c): their schools adopt a mixed approach to placement, which is based on age as well as the result of some type of evaluation of the child's abilities. Six participants reported that students are placed according to age only, while placement in two schools follows the result of a test and/or interview.

Participants who did not choose (a) in Q3 (n = 12) were invited to answer Q4, Q5, and Q6. Q4 asked respondents to describe the instruments used to place new students. A combination of interview and a written test was mentioned by five respondents, while three participants answered that their institutions placed new students after an interview only. Another two respondents stated that new students took a test ("a little evaluation"; "a vocabulary and reading test"). Two participants mentioned a combination of age and a survey filled out by parents, including one who explained that very young students (2–4 years old) were placed according to age only.

Next, Q5 sought to elicit who is in charge of assessing the new students, and how long that assessment takes (be it an interview, a test or a combination of both). Six respondents said that either the school director or a pedagogical coordinator assessed new students, while five participants answered that the teacher was responsible for that assessment (two of those respondents mentioned that such assessment was carried out in the first few classes). One participant wrote simply "according to each student's needs." One participant who had checked (a) in Q3 (new students are placed according to age only) did answer Q5 and explained that they "assess children's progress every class", and if they notice that a child is mature for his/her age and is more advanced than their class, they speak with the parents and, if everyone agrees, the child is moved to another level (provided there is not a significant age difference between the child in question and those in the more advanced level). As for how long the assessment to place new students takes, only six participants specified the duration, which ranged from 10 minutes to four classes (or one month, since classes take place on Saturdays), including 15 minutes, 30–45 minutes, 60 minutes on average, and one class.

Still following up on placement of new students, Q6 addressed when it took place and when and how parents were informed of the decision about placement. Participants indicated that the assessment for placement either took place before classes started (n = 3) or was carried out in the first or first few classes (n = 3). The other six respondents did not provide an answer regarding when placement happened. As for communicating a decision to parents, most respondents (n = 9) indicated that parents were informed of their children's placement, either in a meeting or conversation (n = 7), or through a copy of the evaluation (n = 2). One participant did not specify how the parents were informed. One respondent stated that "(it) is information that pertains to the school only", and it would only be shared if parents demanded it (though it is not clear what "it" may refer to). The other two respondents did not specify how parents were informed about placement decisions.

Results regarding placement procedures address the first research question in this study. Unsurprisingly, we find variation in approaches to placement among Brazilian Portuguese HL programs. Student placement may be done based on age alone, or may involve other elements, such as interviews with parents and/or with students, a type of test, a few classes, or a combination of elements (e.g. interview and test; age and survey filled by parents). When placement is based on more than a child's age, the process (interview, test, survey, class) may take from 10 minutes to four classes and may be carried out by a program director, a pedagogical coordinator, or a teacher.

Q7 in the survey sought to address the second research question in this study. It required an answer, was open ended, and looked for information on assessment during the academic year: whether it was carried out, what instruments were used, what kind of feedback was provided to students and/or parents. Q7 also requested examples of assessment and/or feedback, if possible. Every respondent answered the question, although most did not address all the points in the question. Most participants (n = 12) mentioned that assessment was ongoing and was done by means of in-class or homework activities, which provided the teacher with information about student development. Two respondents (including one who had mentioned reports) mentioned tests: "a type of exam" and "written and oral tests". As for feedback, four participants alluded to a report: three specified that the report was sent to parents, while one respondent wrote simply "a descriptive report", without further details. Five respondents indicated that teachers and/or coordinators spoke with parents about their children's progress. The answer provided by one participant did not shed light on assessment instruments or feedback: "[a] meeting with moments of integration". Importantly, one of the answers that mentioned that assessment was done through in-class

activities also stated that the teacher and the coordinator analyze student performance, and "new strategies are implemented if students don't reach the learning goals" established for the activities. This points to formative assessment, since teaching strategies are adapted depending on how students do in the activities proposed.

The results for Q7 address the second research question in this study: whether (and how) Brazilian Portuguese community-based HL schools assess learner progress during the academic term. For the most part, results of in-class activities appear to provide teachers with the information they need about learner development during the term. In some cases, there may be more formal ways of assessing learner progress, even if the tests used are not considered to carry the kind of formality normally associated with them. One participant indicated that lesson plans may respond to students' needs as evidenced by in-class activities.

Like Q7, Q8 also required an answer. This multiple-choice question addressed summative assessment, asking whether students were evaluated at the end of the academic term. The options were (a) yes; (b) no, because we assess and provide feedback during the term; (c) no, because groups move together regardless of results; and (d) other. Ten participants answered (a), six chose (b), one chose (c) and one responded (d) ("The teacher evaluates students' development during the whole year"). Participants who answered (a) and (d) were asked to also answer Q9, which asked respondents to explain how assessment is carried out at the end of the academic term. However, one of the participants who answered (a) in Q8 did not answer Q9. The results for Q9 show that, for the most part, assessment at the end of the term is not done formally: only three participants mention some kind of test. One respondent wrote the word "exam" in quotation marks, which suggests that s/he does not consider the end-of-semester exam to be the type of formal exam to which s/he may be used. Two participants mentioned that the children do a presentation at the end of the academic year. Another two referred to observing development through in-class activities, while one alluded to a "descriptive evaluation" of each student. Interestingly, two respondents indicated that classes generally do move together (even though they chose option (a) in Q8, not option (c)), with possible exceptions of children skipping a level or being held back.

The third research question that guided this study is addressed by the results of Q8 and Q9. We see that at least one program among those that participated in the survey chooses to move students together, regardless of possible achievement (incidentally, the answer Q7 did not suggest that there was any form of assessment during the term either). In other cases, programs either carry out

### Gláucia V. Silva

some form of assessment (such as presentations or even tests) or rely on information gathered during the term only. The next section turns to a discussion of these results in light of the literature. It also presents suggestions of assessment approaches that can be utilized by Brazilian Portuguese community-based HL programs and possibly programs in other less commonly taught languages.

### **6 Discussion and suggestions**

The data gathered reveal that it seems to be common for Brazilian Portuguese community-based programs to invite parents of new students for a conversation, mostly to explain logistics and to have an idea about whether the child uses Portuguese, and if so, how much. However, other objectives were also revealed in the answers, suggesting that administrators and teachers may view the organization as more than a language school. Raising parents' awareness of the importance of the Portuguese language for their children is an important goal that this conversation may serve (Boruchowski 2014), as revealed in the data. The relevance of bilingualism/multilingualism cannot be overstated; however, in the United States, many still see bilingualism as a problem rather than an asset. As Foster (1982: 342) puts it, "bilingualism is seen by many as evidence of insufficient assimilation." Due to this ideology and the "monolingual bias […] that views bilingualism […] as something that should be eliminated" (Block 2007: 67), parents may be unaware that they need to use the Portuguese language at home if they want their children to learn it. As Lico (2015: 224) notes, the role of community-based schools is "not to make up for or 'fix' what is not done at home; after all, the basis of this process is to recognize and to value the [children's] heritage, of which the parents are the source" (my translation). Thus, some school administrators and teachers feel the need to explain to parents that their use of Portuguese at home is essential for their children's linguistic and cultural development.

Besides the role of the family in maintaining and developing children's HL, the role of the school in the community may also be the topic of the conversation with parents, as mentioned by one of the participants. Community-based schools may organize activities around Brazilian traditions (Dias de Quadros 2017; Souza 2017), providing the diasporic community with an opportunity to gather, meet, and celebrate their traditions. These schools may also teach Portuguese as a foreign language to adults and teach the majority language to speakers of Portuguese (Godoy & Litran 2017), as well as invite the community at large to discuss bilingualism and its advantages (Lira 2017). Thus, the role of community-based schools may go well beyond helping children maintain and develop their HL.

### 6 Assessment in community-based heritage language programs

Although Brazilian Portuguese community-based programs may do much more than teach language, linguistic and cultural development is undoubtedly their main objective. As discussed in §2 and §3, placing students in HL classes is a challenge, be it in university-level courses or in community-based schools. Placement methods elicited by this study vary: while one-third of respondents stated that their schools place new students only according to age, many schools adopt more than one criterion to determine what class a new student should join. Aside from age, interviews and/or tests may be used to place students, as well as surveys that parents fill out. The assessment related to placement may be conducted by a teacher or by someone who fulfills another role, such as the program director or a pedagogical coordinator.

Differences in placement procedures are to be expected, as evidenced by the literature reviewed in §2. While age is a very important factor in grouping children, HL programs should also consider a child's linguistic and cultural skills, as some already do. If a school has enough students to warrant more than one class per age group, then even children as young as four years old can be placed according to their ability in Portuguese. Some suggestions regarding university-level placement exams can be useful for community-based HL programs as well. Fairclough (2012a) recommends that receptive, productive, and creative areas be measured. In the case of children, listening and speaking skills would be assessed; reading and writing may apply in the case of older children, to determine whether they have already developed some ability in those domains. However, it is important to focus on language use, not exclusively on metalanguage (i.e., names of linguistic structures or grammatical terminology). Furthermore, children who learn Brazilian Portuguese as a HL are bilinguals and may display linguistic behaviors that are common among that population, such as code-switching, which demonstrates bilingual competence (Carvalho 2012). However, it is important to keep in mind that a bilingual is not two monolinguals in one and does not develop identical strengths in both languages (Valdés 2001). Therefore, it is not realistic to assess HL learners as if they were monolingual speakers of Brazilian Portuguese; this fact needs to be considered in the development of placement tools. Instead, placement instruments should reflect the local linguistic context, as suggested by MacGregor-Mendoza (2012), so as to assess learners' abilities in their local circumstances.

The instruments used for placement purposes should be developed by each program, taking into consideration the linguistic and cultural experiences of the group these programs serve (Beaudrie & Ducar 2012; MacGregor-Mendoza 2012; Vergara Wilson 2012; Son 2017), as well as the mission of the program (Fairclough 2012a). There should also be ongoing development of assessment tools to respond

### Gláucia V. Silva

to learners' needs and to improve the placement instruments (Beaudrie & Ducar 2012). Furthermore, the data gathered should inform the curriculum of the program and the syllabi for each class (Carreira 2012b; Ilieva & Clark-Gareca 2016). Most community-based HL programs may not have the means to implement electronic placement tests, which would make data compiling easier (Fairclough et al. 2010), so it is important to take care to develop placement tools that do not make data gathering a cumbersome process. While programs should consider their own context in devising these tools, each program should not need to "reinvent the wheel." On the contrary: community-based schools should exchange best practices in order to find out if strategies adopted by other programs may be applicable to their own. Institutions such as universities and consulates may sponsor periodic events geared towards the exchange of best practices and invite representatives of community-based programs in their regions.

Assessment of student progress during the academic term can also vary quite a bit among community-based schools, as attested in the literature. Some of the schools included in Bradunas & Topping's (1988) report tested their students regularly, whereas others left it to the teachers to monitor pupils' progress, and others had informal types of assessment (such as spelling contests). The survey conducted for the present study reveals that, today, Brazilian Portuguese community-based schools also adopt different strategies regarding assessment of progress during the academic term. Most participants in the survey stated that their schools assess student progress during the year, which is done with inclass or homework activities. However, there were no details provided regarding the types of activities that may elicit evidence of development. Two participants did mention tests but did not provide specific information about the structure of such tests. Only one participant mentioned that student performance is analyzed and that new strategies are adopted depending on whether learning goals were reached. This type of analysis suggests that at least some schools may adopt formative types of assessment that allow for instructors "to adapt their teaching so as to attend to the needs of all learners" (Carreira 2012a: 115). As Carreira (2012a) emphasizes, formative assessment helps instructors address issues of learner diversity, which are undoubtedly present in any HL classroom. Even though the examples provided by Carreira are from a college-level class, some of the activities may be used in community-based programs that meet only once a week. Exit cards can quickly provide information about the day's lesson. For example, children who can already write may be asked to list new words learned, or to form a sentence with the structure practiced, or to give feedback on a game. Younger children may be asked to draw words they have practiced, or to color a certain number of squares, or to use certain colors – always receiving instructions in

### 6 Assessment in community-based heritage language programs

Portuguese. Children who can write may be asked to keep journals, which they may share with the teacher either in class or electronically. These may be reading journals, as suggested by Carreira (2012a), or simply journals about what was interesting (or not) in a lesson. The teacher can then identify whether students perceive a given activity as effective, whether they can control linguistic structures that have been studied and/or whether they have grasped a certain cultural concept. Lessons would then be tailored to the needs of the learners as expressed in such assessment tools. This process results in differentiated teaching and learning, which, as Carreira (2012a) and Beaudrie (2016) note, is ideal for HL learners. It is also important to keep in mind that assessment should also be differentiated and children should be evaluated in relation to their own achievement (Douglas 2005), not by comparison to monolingual norms.

With respect to summative assessment, at least one respondent mentioned that students move together as a group and no formal end-of-term assessment is carried out. Other answers indicate, however, that some kind of assessment takes place at the end of the academic term in some schools. Nevertheless, end-of-term assessment is mostly done informally, not unlike what was reported regarding assessment during the term. Even when there is a test, little formality seems to be attached to it, as suggested by the use of quotation marks around the word "exam" by one of the participants. This informality is not a negative aspect: formal evaluations would suggest to children that the HL program is just like regular school, an idea that is certain to demotivate students. Given that student recruitment is one of the challenges faced by these programs (Liu et al. 2011), schools need to do what they can to keep students motivated so they will remain in the program.

Presentations by students, which was mentioned by several respondents, may constitute a formal type of summative assessment. At the end of the academic term, several Brazilian Portuguese community-based programs invite parents to a celebration that includes such presentations. This type of activity may be characterized as project-based learning if students are responsible for choosing a topic, researching it and putting the presentation together with suggestions and help from the teacher. Projects should be an opportunity for learners to develop different linguistic and cultural skills. Young children can also be engaged in projects: teachers may, for example, have learners work on linguistic and cultural elements of a given song during the semester, and present the song at the end. Older students might produce a video on the topic of their choice and/or in consultation with the teacher. Other possibilities for projects that learners can work on during the term include posters, a class magazine, and an art show (paintings, drawings, photos, etc.), among other possibilities that would revolve

### Gláucia V. Silva

around the heritage language and culture. Project-based learning may increase motivation and help students learn (Blumenfeld et al. 1991). Though possibly varied in nature, these projects would culminate in a product that can not only be shown to parents, but may also serve as a springboard for the following year's curriculum for each class, even if the group moves together regardless of individual performance. During the project, teachers may take note of aspects that need to be reinforced and/or revisited as well as aspects that have been acquired and may no longer need special attention. Furthermore, projects may give learners an opportunity to engage in community-based activities while integrating their linguistic and cultural abilities (Ilieva 2007). Teachers and pedagogical coordinators may find ideas on project-based learning, for example, in Beckett & Miller (2006) and in Vaca Torres & Gómez Rodrígues (2017), although such ideas would need to be adapted to each community-based program context.

Although each program must to be tailored to its own context, exchanges of best practices can provide new avenues to be explored. Events such as the Annual Community-Based Heritage Language Schools Conference, held by the Coalition of Community-Based Heritage Language Schools (http:// heritagelanguageschools.org/coalition), gives HL educators an opportunity to listen to experts and to discuss relevant issues. However, many Brazilian Portuguese HL teachers cannot attend the annual conference in Washington, DC. Therefore, consulates and universities that have Portuguese language programs may sponsor events that would be more easily attended by HL educators in a particular region. Like the annual conference in DC, these local events would also serve to keep educators current on research that may inform their assessment practices (Ilieva & Clark-Gareca 2016). Another possibility would be to create an online portal with resources for educators, including a forum. These teachers would likely work in similar contexts, which would possibly allow them to more easily adapt strategies and approaches that have worked well in a given program.

### **7 Final remarks**

This study has contributed to discussions of assessment by shedding light on what is practiced by Brazilian Portuguese community-based HL programs. Limitations of the study include the small number of participants and the lack of details about the forms of assessment practiced by these programs. The fact that respondents completed the survey anonymously, which was done in order not to discourage participation did not allow for follow up questions with respondents which might have clarified what their assessment practices consist of. Future research should include interviews with educators in order to elicit more details

about assessment procedures, including the content of the interviews with parents and with students, as well as the content of tests, how they are administered, and at what point in the school year. Conversations with educators may also elicit their views on types of assessment and what they believe may be more helpful to learners in their particular context.

The suggestions offered here are only a starting point. Heritage language educators have proven to be very creative and capable of achieving a lot with limited resources. Given a chance to learn about what other programs do and exchange resources, these teachers and administrators may be able to help their pupils develop their HL abilities more effectively.

### **Appendix A**

Survey distributed to HL educators in community-based programs (translated from Brazilian Portuguese). The author wishes to thank Ana Lúcia Lico for her input on a previous version of this survey.

	- a. Yes (please answer question 2)
	- b. No (please skip to question 3)
	- a. according to age only. (Please skip to question 7)
	- b. according to the result of an assessment and/or interview. (Please answer questions 4–6)
	- c. according to age and assessment/interview (Please answer questions 4–6)
	- d. Other

### Gláucia V. Silva

	- a. Yes (please answer question 9).
	- b. No, because we assess and provide feedback during the academic term.
	- c. No, because classes move together, regardless of results.
	- d. Other

### **References**


6 Assessment in community-based heritage language programs


### Gláucia V. Silva

toklis Aravossitas (eds.), *Rethinking heritage language education*, 20–44. Cambridge: Cambridge University Press.


6 Assessment in community-based heritage language programs


### Gláucia V. Silva


## **Chapter 7**

## **Indigenous languages in higher education: Case studies from the Amazon of Peru**

### Rosa Vallejos Yopán<sup>a</sup> , Fernando García Rivera<sup>b</sup> & Haydée Rosales Alvarado<sup>b</sup>

<sup>a</sup>University of New Mexico <sup>b</sup> FORMABIAP

> This study examines the role of two indigenous languages in higher education in the Amazon of Peru. It looks at efforts to implement Kukama and Kichwa as key components in the teacher training model developed by the Programa de Formación de Maestros Bilingües de la Amazonía Peruana (henceforth FORMABIAP for its initials in Spanish). At present, Kukama and Kichwa are at different points of UNESCO's endangerment scale. Teaching them in higher education is part of a more comprehensive commitment to build culturally and linguistically appropriate education for Amazonian indigenous groups. This chapter documents several components of the FORMABIAP project. We look at a sample of eight FORMABIAP's alumni, five Kukamas and three Kichwas, to investigate and reflect on the overall outcomes of their learning process. In addition, we survey the profiles of the instructors, the structure of the language classes, and their communities of practice. An important finding of this study is that endangered languages can be relearned by heritage speakers in a combination of naturalistic and well-structured instructional settings. We conclude that the assessment of these relearning processes needs to be holistic, going far beyond linguistic proficiency. In the Amazonian context, at the center of it all are language attitudes, sense of cultural membership, and the learners' positioning with respect to the aspirations of their communities and indigenous organizations. Although the general teaching components can be in fact planned for several ethnic groups, the implementation of the proposals and the evaluation of the outcomes must be done individually and by the indigenous group to capture the uniqueness of each sociolinguistic context.

Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado. 2022. Indigenous languages in higher education: Case studies from the Amazon of Peru. In Eva Rodríguez-González & Rosita L. Rivera (eds.), *Language assessment in multilingual settings: Innovative practices across formal and informal environments*, 153–191. Berlin: Language Science Press. DOI: 10.5281/zenodo.6762282

Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

### **1 Introduction**

The present study deals with relearning processes of endangered Amazonian languages in the context of FORMABIAP (more details at www.formabiap.org). The Amazon Basin is characterized as one of the most culturally and linguistically diverse areas in the world, with about 300 languages belonging to twenty or so language families, plus dozens of genetic isolates (Queixalós 2009). Nevertheless, Amazonian languages are, for the most part, both poorly documented and highly endangered. The majority of these languages are spoken by relatively small speech communities, compared to Spanish or Portuguese, the two major languages with which they coexist in highly asymmetrical relationships.

This chapter focuses on two Amazonian languages: Kukama-Kukamiria (henceforth Kukama) and Kichwa. Their territories are located in the *Loreto* region of Peru. Loreto hosts an extreme linguistic diversity, which includes about 23 languages (Solís Fonseca 2009; Queixalós 2009). However, the speakers of these languages represent only a small percentage of the total population in the region. According to census data from Peru's *Instituto Nacional de Estadística e Informática* (INEI 2017), 9.8% of the population in Loreto self-identify as a member of an indigenous group, yet only 6.4% of them declare to have learned to speak in an indigenous language. These statistics illustrate the increasing social pressure to learn, and ultimately shift to, Spanish.

Languages such as Kukama and Kichwa continue to lose ground in daily use. By including indigenous languages as part of a teaching training program, FORMABIAP aims to counter the progression of rampant language loss. However, data regarding the outcomes of relearning these ancestral languages by heritage speakers do not exist. This is particularly important given that these speakers are, or will eventually be, in charge of teaching these languages in their communities. It is often mentioned that due to their limited command in the ancestral languages, heritage speakers lack the ability to teach these languages (cf. López & García 2016: 127). However, no study has ever attempted to explore how much heritage speakers have actually achieved in the process of relearning their languages. Importantly, in the context explored here, this process does not end at their graduation from FORMABIAP, but continues as they teach the languages to children and engage with other speakers in the linguistic and cultural reanimation of their communities.

This document aims to start a conversation regarding both heritage language teaching and assessment in the Amazonian context. We look at original empirical data collected from eight heritage speakers and offer the first attempt to understand this relearning process in higher education. An additional aim of

### 7 Indigenous languages in higher education: Case studies from the Amazon

the paper is to identify patterns of use, as well as areas of improvement, in the speech of heritage speakers in order to contribute to curricula development. An important finding of this study is that all the participants exhibit various degrees of language expertise. Thus, indigenous heritage languages can be relearned in a combination of naturalistic and well-structured instructional settings, as the outcomes of the multifaceted approach implemented by FORMABIAP are both tangible and substantial.

### **2 Local context**

Any formulation of pedagogical proposals requires a prior characterization of the sociocultural and linguistic contexts of the communities and of the students themselves. In this section, we set the background for the study.

### **2.1 Kukama**

The Kukamas live in small villages spread along several Amazonian rivers. According to the parameters provided by UNESCO (Moseley 2010), the Kukama language is severely to critically endangered. The total number of Kukamas is estimated at 20,000. However, the INEI 2017 census indicates that only 1,185 individuals have learned to speak this ancestral language, and 82.6% of them live in the Loreto region.<sup>1</sup> Most of the remaining fluent speakers are elders, and certainly none of them are monolingual in Kukama (Vallejos 2016b). Children are no longer learning the language, as natural processes of language transmission have been interrupted more than five decades ago. Over the years, a growing concern with respect to the vitality of the language has become evident within the speech community itself, which has resulted in a number of revitalization initiatives. These efforts are being implemented mainly through formal education. As a result, it is possible to find different types of speakers of Kukama, in the sense of Grinevald & Bert (2011), including traditional speakers, latent speakers, neo-speakers, and rememberers (Vallejos 2016b).

One of Kukama's most salient typological features is a morphological distinction between male and female speech in several grammatical categories. Major grammatical categories like person, number, tense, and modality are conveyed by positionally-fixed clitics. Five tense clitics encode three degrees of distance into the past and two into the future. There is a four-way epistemic modal system

<sup>1</sup>According to (INEI 2017), 9.7% of the 1185 Kukamas live in Lima/Callao, 3.8% in Ucayali, and the remaining 4% are spread throughout the country.

### Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

encoding via second position clitics. In Kukama there is neither case marking nor affixal indexation of core participants to express grammatical relations. In main clauses, SVO is the pragmatically unmarked constituent order. While OSV is employed with salient objects, SOV is only possible if the verb is marked by progressive aspect. Oblique phrases are marked by postposition, and clauses are linked by dedicated adverbial subordinators. Clause nominalization is a central subordination strategy, particularly in relativization.

### **2.2 Kichwa**

The Kichwas live in the basins of the Napo, Putumayo, Pastaza and Tigre rivers, in the Loreto region in Peru. Those who live in the basins of the Napo and Putumayo rivers call themselves *Napu Runas* and those who live in the Pastaza and Tigre rivers call themselves *Ingas*. They are also located in the San Martín region, known as *Lamistas*, and in Madre de Dios region, known as *Santarrosinos*. According to the last census, 99% of the Kichwa live in Loreto, and only 1% live in other regions. The Kichwa speakers who live along the Napo River are the descendants of Ecuadorian Kichwa populations called *Quijos* (Mayor & Bodmer 2009). The Kichwa themselves assert that their parents or grandparents came from Ecuador, and a group of them migrated from the Napo River to Madre de Dios during the time of rubber extraction. Although the Ministry of Education of Peru mentions that Kichwa is a variety of the Quechua language, in the last national census (INEI 2017), Kichwa is listed as a different language than Quechua. According to this census, 5,751 declared that Kichwa is the language in which they learned to speak. Although the number of Kichwa speakers in Peru seems to be very small, their linguistic kinship with the Quechua-speaking population (3,799,780 according to INEI 2017) relativizes this situation. Kichwa exists in a range of sociolinguistic situations, from communities where this language is the main means of communication among all generations, to communities where Spanish has almost completely replaced Kichwa (see §2.4).

The Kichwa varieties are part of the Quechua linguistic family. There are, however, important cultural and linguistic differences between Kichwa and Quechua. From a cultural perspective, due to their geographical location in the Peruvian Amazon, the Kichwas' worldview is closer to other Amazonian peoples, including the conception of territory as a fundamental element of life. Perhaps the most salient phonological difference between Kichwa and Quechua is that Quechua makes a distinction between post velar /q/ and velar /k/, Kichwa does not make this distinction. Regarding its typological profile, Kichwa is characterized as an

### 7 Indigenous languages in higher education: Case studies from the Amazon

agglutinating language with suffixal morphology. The preferential order of constituents is SOV, although SVO, OVS are also possible. Grammatical relations are encoded through case marking and verb indexation. Finite verbs take TAM suffixes. Adnominal possession is expresses via a genitive marker suffixed to the possessor.

### **2.3 Speaker types in endangered language contexts**

Amazonian languages exhibit different degrees of endangerment. Speech communities of endangered languages typically comprise several types of speakers who show variance in competence. Grinevald & Bert (2011), building on Dorian (1980), propose four clusters of variables to identify types of speakers: (i) language competence cluster (level of acquisition attained and degree of individual loss), (ii) sociolinguistic cluster (vitality of language at time of acquisition and the age of the speaker at the time of exposure to the language), (iii) performance cluster (level of use of the language and the attitudes towards the language), (iv) self-evaluation of language skills (which can range from under-evaluation, insecurity and self-denial to over evaluation). The implementation of these four parameters gives the following types of speakers (Table 1).

FORMABIAP trains neo-speakers, who, if we highlight their cultural ties to the Kukama and Kichwa ethnic groups, are considered *heritage speakers*. These students enter the program with very limited knowledge of their ancestral languages (more details in §5 and §6).

### **2.4 Sociolinguistic contexts**

Concurrent with different speaker types, Amazonian languages are spoken in a variety of sociolinguistic contexts. Considering the social dynamics and the actual use of the indigenous languages and Spanish, it is possible to identify a wide range of complex scenarios. For example, if we take the Kichwa villages along the Napo River in Peru as a reference point, we can identify a continuum in terms of the vitality of the Kichwa language. In the High Napo River (*Alto Napo*), an area close to the Peru-Ecuador border, there are villages such as Chingana and Humandi where children are learning Kichwa as a first language. In those villages, Kichwa is still the dominant language among all the generations, and very few know any Spanish.

In the High Napo region, but further south from the Peru-Ecuador border, there are villages such as Angoteros and Campo Serio where bilingualism in

### Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado



Spanish and Kichwa is gaining ground among male adults. In those villages, Spanish is used in rather limited contexts; note that by the time children start schooling around the age of five, they are predominantly monolinguals in Kichwa. Even further south in the High Napo region, there are villages, such as Ingano Llacta, where bilingualism in Kichwa and Spanish has been extended to the entire population. In these villages, every generation is bilingual. As a result, children are currently learning both languages from birth.

In the Mid Napo River (*Medio Napo*), in villages like Copal Urcu, San Carlos, Rumitumi, Diamante Azul, and San Jorge, the majority of the children and the youth already speak Spanish as a first, and mostly only, language. However, these generations are still exposed to Kichwa during their interactions with their parents and grandparents. The generations of adults and elders communicate among themselves in Kichwa, but they tend to address their children in Spanish. As a result, most of the children and the youth can understand Kichwa, but show very limited production in this language.

In the villages located along the Low Napo River (*Bajo Napo*), the situation is more pressing. In villages such as Lagartococha, Puca Barranca, and San Juan de Yanayacu, Kichwa has been displaced almost completely. Although it is still spoken by elders, and they may use it in sporadic situations among themselves, Spanish is already the only language of daily communication for all generations. Beyond the Napo River, the Kichwa villages located along the Pastaza River and the Tigre River also show displacement of Kichwa in favor of Spanish, mainly among children and the younger generation.

As for the Kukama villages, most of them have experienced language shift in favor of Spanish, similar to the last scenario discussed for Kichwa. Nowadays, Kukama is only known by elders, and they use it in very restricted contexts. However, there seems to be a difference between the Kukamas and the Kichwas. The Kukamas are more aware of the loss of their language, and in the last decade they have started to express their concerns more openly. In some regions, there are ongoing movements of cultural re-appropriation to reconnect with their roots, and relearning their heritage language is part of this movement.

Note that the possibilities for re-learning the indigenous language decreases from one scenario to another. Taking Kichwa, again, as a reference, it is possible to say that natural language transmission is almost guaranteed in the first scenario. The second and third scenarios result in different degrees of bilingualism; these new language practices tend to go hand in hand with negative attitudes towards the indigenous language. In cases like this, children learning the indigenous language as a first language is no longer guaranteed. In the fourth scenario, children are not acquiring the indigenous language as a first language, but the context is favorable for re-learning it with heritage language methodologies. In the fifth scenario, the functionality of the indigenous language is extremely limited, and so is the possibility of learning the indigenous language without a wellthought-out revitalization effort.

Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

Notably, one of the many challenges FORMABIAP faces to teach indigenous languages is that its students come from all these different scenarios.

### **3 The FORMABIAP Program**

FORMABIAP began its activities in 1988, as a result of an agreement between the Ministry of Education of Peru, the *Corporación Departamental de Desarrollo de Loreto* (CORDELOR), the *Asociación Interétnica de la Selva Peruana* (AIDESEP), and the Italian NGO Terra Nuova. FORMABIAP was created as a program to train indigenous teachers in intercultural bilingual education (IBE) within the framework of a Pedagogical Higher Education Institute, based in the city of Iquitos. From its inception, FORMABIAP was a pioneer model for a genuine collaboration between the government and a national indigenous organization. The role of indigenous specialists and scholars in the teacher training process, as well as the participation of community members in the development of the curricula for elementary education are some of the key components of its success. In its 32 years of work, FORMABIAP has trained 1,213 indigenous teachers from 16 Amazonian ethnic groups, who have taught an estimated 363,900 Amazonian children (Trapnell Forero et al. 2018).

FORMABIAP considers that languages and education must be approached from the specific social contexts in which the project operates, and from a political and historical vision that allows its students to understand how the processes of colonial domination have influenced the current situation of the indigenous languages (Trapnell Forero et al. 2018: 38). At the beginning of the FORMABIAP project, most students had an indigenous language as their L1, with the exception of the Kukamas who were mostly monolingual in Spanish. Over the years, the sociolinguistic landscape has become more heterogeneous and complex to address. On the one hand, there is now a greater number of students with serious limitations in oral Spanish. On the other hand, the number of students that have Spanish as a first language has increased.

### **3.1 Why heritage language instruction in FORMABIAP**

A second language is any language an individual learns in addition to their first language, and no previous knowledge of this language is implied. In contrast, a heritage language entails certain exposure to the target language during childhood, as well as a cultural connection to the language through family, ancestors, community, or country of origin. In this view, a heritage language can be an

### 7 Indigenous languages in higher education: Case studies from the Amazon

indigenous language that underwent displacement (i.e. Kukama and Kichwa in Peru) or an immigrant language (i.e. Spanish in the United States or Turkish in Germany). In a narrow definition, heritage learners are bilinguals, but the language they learned in childhood at home is no longer their dominant language; instead, they are dominant in the hegemonic language of the society in which they live. In a broad definition, heritage learners have had at least some input during childhood but did not grow up with this language as a means of communication (Polinsky & Kagan 2007; Rothman 2009). As a result, heritage language learners can display widely diverse levels of proficiency (Valdés 1989, 2005). In the cases discussed in this article, we adopt a broad definition of heritage learners.

Heritage languages, as all languages, are surrounded by ideologies, from issues of identity and group membership, to views of language as a commodity that question the value of an ancestral language in current times. Other common ideologies relate to who are the "true owners" of a given language, who should have the ultimate say on how the "real language" should sound like, and diverse perceptions regarding language variation and change (see, for instance, Lopez Odango 2015). Thus, in multilingual settings such as the Amazon, the development of intercultural communication competence needs to be stressed. This development involves self-reflection concerning language and culture, and it involves attitudinal changes toward one's own and others' cultures and languages.

The fact that heritage learners bring some linguistic knowledge from childhood has opened important avenues in second language acquisition research, particularly on the role of age, input, and implicit versus explicit knowledge (Valdés 2005). For example, heritage speakers seem to have an advantage over L2 learners not only with regards to phonology, but also in some aspects of morphology and syntax (Montrul 2008). Issues related to linguistic mechanisms and the types of patterns that emerge in language loss and revitalization contexts (Hinton 2001) are also important theoretical questions for language change. The impact of the results of these new lines of research in teaching methodologies continues to be explored (Parodi 2008, Potowski 2018).

Motivation is critical to language learning, but particularly so in heritage language learning. If cultural heritage is a motivating factor to learn a language, it can promote learner autonomy to continue the learning process beyond the formal settings. This is critical in endangered language contexts where the target language is no longer used in daily communication. For example, leaners of Kukama and Kichwa have to make an effort to build a community of practice in their villages.

Heritage learners can also vary with respect to forms of exposure and age of acquisition (naturalistic process during childhood, formal instructional settings as adults, revitalization contexts guided by elders and activists, etc.), communities of practice, and amount of access to input in the target language. The heritage speakers of Kichwa and Kukama who participated in this study were exposed to their ancestral languages to various degrees during childhood; however, all of the participants manifest a strong cultural connection and a sense of membership to their respective ethnic groups.

### **3.2 Profile of learners**

Three decades ago, aspiring Intercultural Bilingual Education (IBE) teachers were older than the average high school graduates in Peru, and many had not completed a secondary education. Most of the students at the time predominantly spoke an indigenous language as their first language, with the exception of the Kukamas. In more recent years, however, the completion of a high school education became a prerequisite to enter FORMABIAP. By going through the complete educational system (i.e. kindergarten, primary, and secondary), in many cases outside their villages, these students have acquired greater proficiency in Spanish than those that entered FORMABIAP 30 years ago. Nevertheless, an unintended consequence is that these new generations have had fewer opportunities to learn the cultural knowledge of their people and their respective indigenous language. As a result, the most recent cohorts of students who come to FORMABIAP show less command of their indigenous languages. That is, the linguistic profile that used to be associated with the Kukamas has now been extended to other indigenous groups, such as the Awajun, Shawi and Kichwa. Note, though, that the Kukamas and Kichwas come with a very limited command of their heritage languages not only because of the formal schooling, but also because the generation of grandparents who speak those languages is declining rapidly.

Under these conditions, the relearning of an indigenous language by the future teachers is even more challenging. But even for students who have an indigenous language as their L1, having had less exposure to that language in the community life entails that it needs to be taught and expanded at the oral and written levels. In several cases, the indigenous language was generally used only in primary education (K-5), and then its use was interrupted at the secondary level. In the Amazon of Peru, secondary education is developed almost exclusively in Spanish. It is then imperative that the new cohorts that join FORMABIAP strengthen their competence in their indigenous languages, as well as their cultural traditions.

### 7 Indigenous languages in higher education: Case studies from the Amazon

However, the demand for learning the indigenous languages arises not only because of FORMABIAP's institutional mission, but because there are now national IBE policies that demand communicative competence of the indigenous languages in the schools that provide an intercultural bilingual education (Ministerio de Educación del Perú 2016). In addition, according to the new official policies, teachers working in communities with heritage indigenous languages must implement cultural and linguistic revitalization (Ministerio de Educación del Perú 2016). In those communities, children have access to very few opportunities to listen to the indigenous language because it is spoken only by older people in very restricted contexts. Thus, given that teachers have to teach these heritage languages to the children of these schools, the indigenous language must be relearned as part of their teacher training in FORMABIAP.

### **3.3 Profile of instructors**

FORMABIAP hires language and culture *specialists,* who are fluent elder speakers. They come with their partners to live in Zungarococha, together with the students. Because of this configuration, these elders are mentors and make themselves available to the students almost at all times and are a crucial resource on every aspect of their training.

Second, the *instructors* of the indigenous languages are community members with training in language teaching methodologies. The *instructors* are in charge of preparing and implementing the lesson plans with the input from the elder*specialists*. In addition, in some cases, there is also a *linguist* who collaborates with the specialist and the instructor. The role of the linguist in FORMABIAP is more focused on developing awareness about the structures of the language and guide the teaching process. They participate in the development of teaching materials, the incorporation of new genres and discourse practices into the classroom, and overall literacy development issues.

The Kukama and Kichwa participants of this study had an elder specialist and one or two instructors. Sporadically, the cohort also included a linguist in charge of the grammatical study of the indigenous language. Within the Kichwa group, there were also students who were fluent in Kichwa in the same cohort. For instance, this provided participant kich-3 with opportunities for practice the language with his peers.

Beyond this formal training in FORMABIAP, heritage learners have the opportunity to further develop their linguistic skills during their professional practices in the villages. These are important periods in which heritage learners immerse

themselves with other speakers of the languages, and the confrontation with their own communicative skills is unavoidable.

### **3.4 Language classes**

Six months per year, during five years, FORMABIAP students from different ethnic groups come to live in Zungarococha, a sort of academic town located away from the Iquitos city center. During this time period, FORMABIAP tries to guarantee that the students have different types of support to relearn their heritage languages. The teacher training curriculum has always considered a space for formal classes and hands-on workshops around the indigenous languages during their five years of study. The classes and workshops are conducted on average two hours a week, during nine semesters. For those students that do not need to relearn the indigenous languages, the workshop is aimed at the development of new discourse practices, literacy development and reflection on the grammar of the language. For heritage speakers, such as the Kukamas, these spaces are designed to relearn the indigenous language with the help of an instructor, a specialist, and possibly a linguist.

However, indigenous languages in FORMABIAP are not reduced to specific classes. They have a fundamental role in several other curricular components. For example, there is an umbrella educational project planned for each year of the training that is implemented by each indigenous group, rather than by academic level. Past projects include: technologies of house building, natural dye techniques, practices of traditional healing, indigenous food and diets, etc. Under these projects, topics from different areas (such as mathematics, social studies, ecology, communication, pedagogy) are developed through integrated activities. The indigenous languages are the language of communication during all those activities. Thus, these spaces constitute additional opportunities of exposure to the languages for heritage learners. However, it is also important to underscore that several students show high motivation to learn the language and so constantly seek to create new support mechanisms for their learning. For example, taking advantage of the technology, most of them now have a dictionary and/or a translator on their smartphones.

To monitor the progress of the students, FORMABIAP administers a diagnostic test to every student at the beginning of the five-year program, multiple assessment strategies throughout the nine semesters, and, in some cases, an achievement final test with written and spoken components. It is important to consider that, according to FORMABIAP's guiding rules, heritage speakers need to show systematic progress for their scholarships to be renewed. In fact, some students

### 7 Indigenous languages in higher education: Case studies from the Amazon

have been removed from the program given lack of progress in this area. The discussion of these details is beyond the scope of this article.

Having set our framework and the background, in the next section we introduce the empirical methods employed to measure proficiency in a sample of Kukama and Kichwa alumni.

### **4 Assessment: approach and instruments**

In general, heritage languages are also less commonly taught languages, or are only taught in non-mainstream environments. In the cases of Amazonian languages, few culturally appropriate materials for higher education exist, and setting baselines and benchmarks to measure proficiency in heritage speakers is still an area in need of attention. In addition, there is controversy whether the proficiency guidelines employed to assess second language learners is appropriate for heritage language abilities (Valdés 1989, Martin et al. 2013). Assessment of attained proficiency among heritage learners is essential for understanding the characteristics of their speech. It is crucial to identify areas of strength and error patterns in order to inform the curricular needs of instructional programs. Above all, it is important to understand heritage students' motivation to learn languages that "lack" prestige and functionality in the eyes of the dominant society.

A basic assumption here is that language proficiency is a continuum.<sup>2</sup> We follow an Integrated Performance Assessment approach to survey the performance of individual students and determine an overall, "big picture" view of their progress (ACTFL 2012). We employ Can-Do Statements, which require learners to show what they know and can do, to demonstrate their language proficiency. Can-Do Statements reflect the continuum of growth in communication skills through different levels (Moeller & Yu 2015). Our goal was to examine the participants' ability to use culturally appropriate language, as judge by language consultants, to communicate spontaneously in non-rehearsed contexts. But encouraging heritage speakers to produce spontaneous speech, or even eliciting narrative using stimuli, can be difficult given the surrounding social pressures and ideologies that can derive in linguistic insecurities (see, for instance, Silva-Corvalán 1994). In order to collect empirically based data, we provided learners with opportunities to use their heritage language in authentic, real-world situations. We used an authentic video, picture cards, and a printed questionnaire.

<sup>2</sup>According to the American Council on the Teaching of Foreign Languages Guidelines (ACTFL 2012), the continuum consists of four major levels: novice, intermediate, advanced, and superior. These major levels are further subdivided in eleven levels. The ACTFL Guidelines are intended to be used for holistic assessment and have instructional implications.

Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

### **4.1 Video**

Videos were used for comprehension tasks. For Kukama, we selected a culturally relevant video of a procedural text collected within the context of the Kukama-Kukamiria Language Documentation Project (Vallejos 2014). It shows a 65-yearold woman explaining how to make a hand fan with a palm tree leaf. Only the first 10 minutes of a 17-minute video were selected. In this excerpt, the speaker introduces the materials and tools to be used, and then proceeds to describe a sequence of steps to make a fan as she weaves one. This video is particularly useful because of the pace of the speech as the speaker is carefully explaining the process.

The participants watched the video twice. In the first pass, they completed a word recognition task. They wrote down all the words and expressions they could identify as the video was playing. In the second pass, they focused on both main ideas and supporting details and captured them in phrases and complete sentences. The second pass was also an opportunity to check and augment their word lists.

### **4.2 Picture cards**

Picture cards were employed for production tasks. Two sets of picture cards that portray Amazonian scenes were selected as visual stimuli to elicit guided stories. These cards are part of the series *Cuentos en Tarjetas* created by Amías et al. (2003), from FORMABIAP. The first set consists of cards that portray scenes about a dog that goes hunting with its owner in the jungle (see Figure 1). The second set consists of cards about three armadillos who go out to collect wild seeds and, while doing so, fall into a trap.

The participants organized the cards in the order of their preference and then proceeded to create a story. These stories were recorded to allow qualitative and quantitative analyses, including speech rate. These speech samples are rich in objectively recognizable linguistic features in the areas of phonetics, phonology, morphology, syntax, discourse, and lexicon. It is important to note, however, that this procedure elicits samples of what speakers can do and help us identify areas of improvement, but it is not necessarily useful for discovering areas that speakers do not know yet. In the future, we may need to include other tasks to capture what advanced learners cannot do and need focused help with.

7 Indigenous languages in higher education: Case studies from the Amazon

Figure 1: Example of a set of picture cards (Amías et al. 2003)

### **4.3 Questionnaire**

We created a questionnaire to collect biographical information to determine what social and cultural factors influence their attitudes, motivations, and linguistic choices (cf. Alarcón 2010). It includes 32 items in total. The political and affective issues surrounding heritage speakers came to light in these questionnaires. The first portion elicits information regarding their exposure to their ancestral languages during childhood, their motivations to become bilingual teachers, and their knowledge of their ancestral language before coming to FORMABIAP. In the second part of the questionnaire we collected information regarding language behaviors, attitudes towards dialectal and generational variation, and selfassessments of their heritage language abilities. Data from the questionnaire allows the examination of potential correlations between self-reported proficiency level, and speech rate, word recognition, and the use of specific grammatical patterns.

In the next section, we present preliminary findings on linguistic correlates of proficiency in Kukama and Kichwa as heritage languages, based on data from the questionnaires, as well as perception and production tasks.

### **5 Results: Kukama**

This study looks at five Kukamas. They are all heritage speakers that entered the program with very limited knowledge of their ancestral languages. The profiles of the participants in which group is described in detail below.

Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

### **5.1 Participant profiles**

Five heritage speakers of Kukama, four males and one female, participated in this study. Their ages ranged from 24 to 36 (avg = 28), and they all attended FORMABIAP between 2012 and 2016.<sup>3</sup> They had two male instructors of Kukama, who are themselves community members and FORMABIAP alumni (see §3.2). During their training, they were also mentored by four Kukama specialists, two male and two female elder speakers, who worked in the project at different points in time.

Data from anonymous questionnaires show that after finishing high school, they all wanted to become bilingual teachers, and four of the five participants indicated having the support from their communities and indigenous organization. As for their exposure to Kukama at an early age, four of the five participants indicated that their parents understand the language and that their grandparents would speak it, but only from time to time because they lack regular conversation partners. All the participants are aware of the level of endangerment of Kukama and seem committed to ongoing preservation efforts. They display very positive attitudes towards their Kukama identity and their heritage language; however, four of the five participants indicated they prefer not to speak Kukama outside their community contexts to avoid public shame. As for language ideologies regarding language variation, all the participants acknowledge geographic and generational differences; however, four of the five participants indicated they aspire to speak like the elders because they speak the true Kukama.

Regarding their knowledge of Kukama at the time they enrolled in FORMABIAP, all of the participants say they knew common words and expressions. For words, the examples provided in the questionnaire are *ipira* 'fish', *yawara* 'dog' *atawari* 'chicken', *arara* 'macaw', *uni* 'water', *irara* 'canoe', '*yapukita* 'paddle', etc. For expressions, they listed: *era na kuema/karuka* 'good morning/afternoon,' *tsaniuri* 'Come on in', *makatipa na utsu* 'Where are you going', *ta tseta eyu* 'I want to eat', *ta kurata kaitsuma* 'I drink yucca bear'. Even though they knew some words and expressions, all of them said they could not understand and engage with fluent speakers of Kukama. In the self-assessment of their proficiency, all declared having made significant progress in learning the language. On a 5-point scale, they gave themselves an average score of 4.2 for reading and writing, and 3.2 for speaking and listening.

Given the lack of baselines and benchmarks to assess proficiency in Kukama, a fluent, elder speaker with 20 years of experience as the community linguist,

<sup>3</sup>The participants attended a propaedeutic in 2011, before starting their teaching training, but Kukama was not included in this preparatory phase.

### 7 Indigenous languages in higher education: Case studies from the Amazon

and who was an instructor of the Kukama language in the FORMABIAP project, assessed the speech samples of all the participants. She listened to each story and grouped the participants into three categories: A: *está aprendiendo* 'he/she is learning', B: *habla, pero tiene que aprender y practicar más* 'he/she speaks, but needs to learn and practice more', C: *ya habla, pero necesita corregir algunas palabritas* 'he/she speaks already, but needs to fix some little expressions'. We interpret these categories as A being towards the lower end of the proficiency continuum, and C towards the higher end of the continuum. According to this specialist, the participant kuk-1 should be in category A, kuk-2 and kuk-3 in category B, and kuk-4 and kuk-5 in category C. The profiles of the Kukama participants is summarized in Table 2.

Table 2: Kukama participants


### **5.2 Oral comprehension**

The data to assess oral comprehension comes from the activities with the video as explained in §4.1. All the participants completed the word recognition task with extreme ease. Remarkably, four of the five participants listed only those items that they clearly knew; only one participant listed a couple of nonwords, which were excluded from the counting. The vast majority of the items registered were content words, including nouns (e.g. *marawi* 'hand fan', *mɨrɨti* 'palm tree'), verbs (e.g. *imaki* 'select', *kauki* 'wait' ), and adverbs (e.g. *ikun* 'today', *ikumenan* 'soon'); only one participant listed also a few function words, such as pronouns and demonstratives (ex. *ay* 'he/she', *ajan* 'this'). We included both of them in our calculations.

The second task consisted of identifying phrases and sentences. All the participants were able to recognize and isolate a variety of syntactic structures. Within the phrases category we report only those that were listed on their own, not as part of another larger syntactic constituent (i.e. NP objects within a clause

### Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

were not counted as phrases). This category comprises noun phrases (ex. *ini puwa '*our hand'), verb phrases (ex. *uchima tsa* 'extract the leaf') and nominalized constructions (ex. *kuarachi tatatan* 'something that has dried with sunlight'). Interestingly, no one listed postpositional phrases on their own. In the clauses category, we included simple clauses (ex. *ikun kuashi ini yauki marawi* 'today we will make a hand fan'), complex constructions (ex. *awanu tseta purepeta ajan* 'people want to buy this', *ini yaukiai imaki ipukun* 'we make it by selecting the long ones'). The results are provided in Figure 2. Note that only one participant, kuk-1, listed more phrases (n = 9) than clauses (n = 5). All the other participants registered more complete clauses than phrases, which may point towards more advanced comprehension skills. However, there are not significant differences among the participants in the overall comprehension of phrases and clauses.<sup>4</sup> In sum, the data suggests that all the participants have achieved strong comprehension skills.

Figure 2: Kukamas' oral comprehension

### **5.3 Oral production**

As with the comprehension tasks, we let the benchmarks for Kukama oral proficiency emerge from the data itself. The literature on heritage language teaching and learning suggests that heritage learners follow unique trajectories and

<sup>4</sup> 2 : 6.549, : 0.161727. The result for *phrases* and *clauses* is not significant at < 0.05. It becomes significant if we include *words*.

### 7 Indigenous languages in higher education: Case studies from the Amazon

should not be compared against traditional speakers (see, for instance, Valdés 2005). Following this view, the benchmark to measure oral proficiency in this study is not the speech of the elders. In collaboration with speakers of the language, we transcribed the recordings of the stories produced by the participants to quantify several parameters. First, because each participant was invited to speak for as long as he/she wanted, we recorded the length of each story. Second, we quantified the total number of words used. Third, we calculated the number of word types (including both function words and content words) to get a sense of vocabulary knowledge and the amount of repetition of words. Finally, we calculated speaker rates as word-per-minute output by dividing the total number of words by the length of the stories. The idea being that lower proficiency speakers have more difficulty in accessing lexical items, which slows down their speech. The results for both stories are provided in Figure 3.

The results for oral production suggest that participant kuk-1 is at a lower level in the proficiency continuum compared to the other participants, particularly with respect to the length of the stories and the overall number of words produced. Kuk-1's score for word types (n = 55) is also lower than the average for all the participants (avg = 94). However, a chi-square test comparing word types and speech rate reveals that there is not a significant difference among the participants.<sup>5</sup>

Figure 3: Kukamas' word tokens and types in oral production

An interesting finding is that the assessment provided by the Kukama specialist does not align well with some scores in Figure 3. While her observations re-

<sup>5</sup> 2 : 4.4693, : 0.346208. The result for *word types* and *speech rate* is not significant at < 0.05. It becomes significant if we add *number of words*.

### Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

garding kuk-1 seem to hold, according to this specialist, participant kuk-3 should be a little behind participants kuk-4 and kuk-5. Nevertheless, as shown in Figure 3, kuk-3 has the highest scores for total number of words produced and word types employed. Thus, these scores offer only a glimpse into the story of this relearning process. In order to have a fuller picture, and guided by the specialist's observations, we look into specific linguistic features produced by the participants.

### **5.4 Qualitative analysis**

Our results suggest that knowledge of lexical items and speech rate might not be correlated with grammatical knowledge and pragmatic competence. This seems surprising given that lexical access tends to also be accompanied by difficulty constructing phrases and clauses. Thus, some discussion of the results on specific subcomponents of the grammar are in order.

### **5.4.1 Phonetics and phonology**

A recurrent observation about heritage speakers is that even the novice sound native-like, which contrasts with what we see in conventional second languages (Polinsky & Kagan 2007). This holds true for the Kukamas as well. For instance, impressionistically, their intonation patterns sound comparable to fluent speakers. Stress assignment is always on target; it is realized in the penultimate syllable except in words that end in a consonant (ex. *éyu* 'eat', *eyún* 'food'). Phonological processes are consistently implemented (ex. sonorization of voiceless stops following nasals, as in *temente* [temende] 'there is not', *ajanka* [ajanga] 'here'). Optional phonological processes are implemented randomly (palatalization of affricate *tsitsa* [chitsa] 'face'). All participants tend to produce the central vowel /ɨ/, as /i/. Arguably, they have not added yet this vowel to their vowel inventory.

### **5.4.2 Morphosyntax**

In Kukama, grammatical categories such as person, number, tense, and modality are conveyed by positionally fixed clitics. All the participants make use of a subset of these forms. Importantly, given that no suffix or clitic is obligatory in Kukama, the lack of bound morphology do not render structures ungrammatical. The most frequently used postpositions are -*ka* 'locative', -*pu* 'instrument', -*muki* 'comitative', and -*kuara* 'inesive'. Documented verbal morphology include -*ka* 'iterative', -*ta* 'causative', -*ari* 'progressive'. The completive -*pa* was not documented, and the use of past tense markers is also limited. All the participants

### 7 Indigenous languages in higher education: Case studies from the Amazon

also used plural markers and nominalized forms and their underived counterparts. Some examples are *eyu* 'eat', *eyun* 'food' *ipurkari* 'hunt', *ipurkarin* 'hunter'. The focus clitic =*pura* was used by two participants, and generally with the same host which suggests they learned it is a chunk, as shown in (1).

(1) rian=*pura* then=foc ikian dem.ms awa=*kana* person-pl.ms umi see ra 3sg.ms yawara dog 'at that moment these people see his dog'

One of the most salient typological features of Kukama is the presence of grammaticalized gender indexicals (for details, see Vallejos 2015). Kukama does not have grammatical gender; that is, it does not encode the gender of a referent. Kukama's gender indexicality is a categorical distinction that encodes the gender of the speaker. Male and female speech is expressed in several categories, including personal pronouns, indefinite pronouns, demonstratives, number marking, and connectors. Heritage speakers use plural markers, as =*kana* in (1), and the first-person pronouns (*ta* vs*. tsa/etse*) quite accurately. The second-person pronoun, *na*, does not vary from women and men. However, some of the participants tend to have difficulties with third-person pronouns (*uri/ra* vs. *ya/ai*) and, the first-person exclusive pronouns (*tana* vs*. penu*) did not show up in the stories. Although one male participant used a few female forms (*yamua* instead of *ramua* 'other', *yaepetsui* instead of *raepetsui* 'then'), male speakers consistently used male pronouns, as *ra* in (1). The female speaker had more difficulties with gender indexicals, as discussed further, below.

Kukama has several strategies to combine clauses into more complex sentences. Clause nominalization is a central subordination strategy, particularly for relativization functions. The language has a set of subordinators to express several logical relations, such as cause, condition, and temporal simultaneity (Vallejos 2016a). The participants made very limited use of clause combining strategies.<sup>6</sup> To link simple clauses, they use prosody; clauses are produced within a single intonation contour and the semantic relationship between clauses are left to be inferred from context, as shown in (3). This is an area that needs attention.

An area that seems to represent a challenge is information questions. Kukama has the interrogative marker -*tipa* that is attached to an interrogative pronoun, or the piece of information under interrogation. Only two participants attempted to make questions with this morpheme, the others used only rising intonation.

<sup>6</sup>Two of the five participants used two purpose subordinators at once, but with the same verb (*eyu-mira-tsen* eat-pur2-pur3). This sequence may have been learned as a single chunk.

### Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

However, the syntactic structure of the attempted questions tends to be problematic, as shown in example (2a), produced by kuk-5. In (2a), the sentence has an interrogative pronoun, but the interrogative marker is in the verb. Also, the subject of the clause is missing. The Kukama specialist provided two potential target constructions according to the context of the story, which are given in (2b). In the first, the identity of the object is being interrogated. In the second, the predicate is being interrogated, but in this case the object argument needs to be realized.

	- b. mari=*tipa* What=int na 2sg tseta want eyu eat / Tseta=*tipa* want=int eyu-n eat-nzr 'What do you want to eat?' 'Do you want food?'

It should be highlighted that some types of complex predicate constructions – i.e., clause constructions with more than one predicate – are employed by all of them. Some examples are provided in (3c) and (3e), below.

### **5.4.3 Discourse pragmatics**

Recall that according to the scores in Figure 2, the participants could be located at relatively similar points in the proficiency scale, except kuk-1. However, the Kukama expert put them in three groups. kuk-2 and kuk-3 were categorized in group B ("he/she speaks but needs to learn more and practice more"), while and kuk-4 and kuk-5 in group C ("he/she speaks already, but needs to fix some expressions") by the Kukama expert. The explanation seems to lie in the fact that their speech differs in terms of discourse organization.

Story telling is an important cultural practice among Amazonian peoples. Kukama elders are generally exceptional storytellers, and most traditional stories have a message regarding social norms and expectations in the community. These stories are told for the most part in the third person, and most of them concern animals interacting with each other and their surroundings (see an example in Vallejos 2018). Their stories are full of dialogue and direct quotations. To incorporate direct speech from participants assigned to different sex categories, they re-center the referents of the gender indexicals for each speech event. The stories collected with the picture cards lack these features, which is perhaps explained by the artificiality of the stimuli.

### 7 Indigenous languages in higher education: Case studies from the Amazon

Consider (3), an extract from one of the stories produced by participant kuk-4. This story flows well. There is almost null use of bound morphology, but this speaker employs complex predicate constructions, as in (3a), (3c), and (3e), as well as reduplication of verbal roots, as in (3e), to express aspectual subtleties.

	- 3sg.ms go smell
	- d. tsetuni smell ria too animaru, animal hm hm
	- e. ra 3sg.ms yupuni start kari-kari scrape-scrape '(a) It [hunting dog] goes to see one, a house of an animal, (b) he digs, (c) he goes to smell it, (d) to smell this animal's (house), (e) he starts to scrape and scrape' (kuk-4)

An interesting point that emerged in the speech of participants kuk-2 and kuk-3 is the overuse of the second person singular pronoun *na* for impersonal functions. Elder, traditional speakers of Kukama do not use *na* for generic, impersonal reference. In the excerpt in (4), the speaker kuk-2 seems to be describing the activity of hunting, not creating a story about the dog. If we substitute the pronoun *na* 'you' for *ra* 'he/she', we would have a third person story, similar to what we see in (3). In the extract in (5), the participant kuk-3 uses of *na* in similar ways, although in some cases, the resulting constructions are problematic and difficult to understand, as in (5d).

	- b. na 2sg erutsu bring yawara=muki dog=com
	- c. na 2sg chikari look.for wepe one animaru animal
	- d. ikian, this na 2sg papa, father na 2sg

### Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

	- b. umi=ura see=3sg.ms na 2sg tseta want upi=nan all=foc […]
	- c. tima neg na, 2sg na 2sg yumi give eyu-n eat-nzr […]
	- d. upi=nan all=foc tua-n=kana big-nzr=pl.ms titi-ka alone-rei na 2sg eyu-mira eat-pur
	- e. tima neg na 2sg yumi give animaru animal ipurkari-n hunt-nzr
		- '(a) This hunting dog sees it [the food], (b) sees it (but) you want all,
		- (c) you don't share the food, (d) all the adults are alone for you to eat,
		- (e) you don't give to the hunting dog'

But why would these two participants use *na* instead of *ra* in story telling? One hypothesis is because the second person pronoun does not vary depending on the speaker's gender, as does the third person (*ra* vs. *ya*). A second hypothesis is Spanish influence. This impersonal use of *na* resembles the use of Spanish *tú* for similar discourse functions in Amazonian Spanish (Vallejos et al. 2020), as well as in English as evidenced in the translations.

The fragment in (5) is also interesting for other reasons. Speaker kuk-3 uses more bound morphology than other participants (i.e., the plural marker, clitic pronouns, nominalizer, focus, the subordinator of purpose), but recall that this speech was nonetheless rated lower than of kuk-4 and kuk-5. In addition to the overuse of *na*, this speaker mixes gender indexicals. For instance, in (5a), kuk-3 uses *ajan*, the demonstrative of female speech, but in the same line she uses =*ura*, the clitic pronoun for male speech (instead of =*ay*), and in (5d) the plural marker for male speech =*kana* (instead of =*minu*). Traditional speakers tend to be sensitive to the use of gender indexicals. But mastering this feature is difficult, and more so if there is not enough input of both types of speech and opportunity for practice. Note that the instructors of Kukama are males. Hiring female instructors should be considered in instructional planning in the future.

### 7 Indigenous languages in higher education: Case studies from the Amazon

An additional point to note regarding the speech of heritage speakers is the innovative uses of *wepe* 'one'. This cardinal number is used as indefinite determiner, as seen in (3a), (4c), and (5a), probably because of Spanish influence. For example, everyone said *wepe kuashi* 'one day', which would work well if we were counting days, but not to make reference to a point in time in the past. For these function, traditional speakers would use expressions such *ɨmɨnua* 'long time ago'*, yamua/ramua kuashi* 'another day'*, ikun kuashi* 'today', etc.

A final point regarding discourse is the very limited use of code switching by these participants. They all inserted very few loanwords, but nothing that would be considered switches to Spanish. This is surprising since in a previous study, with a different speaker sample (Vallejos 2016b), switching was extensively used by heritage speakers. A possible explanation is that the participants in this study have studied under different instructors.

### **6 Results: Kichwa**

### **6.1 Participant profiles**

Three heritage learners of Kichwa participated in this study: two women and one man. kich-1 and kich-2 are women, and their ages are 30 and 35, respectively. They attended FORMABIAP between 2015 and 2019 to get training as teachers of preschool education (*Educación Inicial Intercultural Bilingüe*) during the summer periods. It needs to be highlighted that the training of preschool teachers is different from the training of elementary school teachers. The former are teachers that must have a teaching position in a preschool to attend formal training in FORMABIAP; the latter do not hold a teaching position prior to graduation. As a result, participants kich-1 and kich-2 have had limited access to structured classes of Kichwa during their time at FORMABIAP; that is, they are mostly learning the language in their villages while working with kindergarteners. The third participant is a 22-year-old man; he attended FORMABIAP from 2012 to 2017 to become an elementary school teacher. As such, he has taken classes of Kichwa during his five years at FORMABIAP.

In the anonymous survey applied, two participants said they entered FORMABIAP because of the scholarship offered to carry out their studies, and because of the support of their communities and families. However, the three of them indicated that their motivation to learn Kichwa emerged in the framework of their professional training in FORMABIAP. kich-3 self-reported that, in addition to the classes at FORMABIAP, he systematically immersed himself with

### Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

fluent speakers in the villages where Kichwa is the dominant language to gain proficiency.

Regarding prior knowledge of Kichwa before entering FORMABIAP, they stated that they knew common words like *challwa* 'fish', *wallpa* 'hen', *yachachikama* 'teacher' and phrases like *allipuncha* 'good morning', *shamuy* 'come', *kuyntaway* 'tell me'. However, they may have comprehended more words and phrases because two of them said they listened to their grandparents speak Kichwa during their childhood. All three said that in their community there were older adults who spoke Kichwa fluently. That is, the three participants of this study were exposed to Kichwa during their childhood by their grandparents and other elders; however, they did not foster the use of the language because the social conditions did not exist.

In the same survey, the three participants indicated they are proud of the Kichwa language and think that the elders speak the true Kichwa and that the young people should learn from them. In summary, the three participants have a strong appreciation of Kichwa, which is a good motivation to continue to learn this language.

A Kichwa specialist assessed all the speech samples to provide some input regarding the overall proficiency of the participants. According to this specialist, kich-1 and kich-2 should be assigned to category B (*habla, pero tiene que aprender y practicar más* 'he/she speaks, but needs to learn and practice more'), while kich-3 is in category C (*ya habla, pero necesita corregir algunas palabritas para hablar fluido* 'he/she speaks already, but needs to fix some little expressions to speak fluently'). That is, kich-3 is the most advanced of the three in terms of proficiency. A summary of the profiles of the Kichwa participants is given in Table 3.


Table 3: Kichwa participants

### **6.2 Oral comprehension**

Given some technical difficulties, we could not collect oral comprehension data similar to the Kukamas. However, two Kichwa instructors who taught the three

### 7 Indigenous languages in higher education: Case studies from the Amazon

participants indicate that all of them display advanced oral comprehension to fully understand narratives and descriptions. For example, in FORMABIAP there are certain sessions conducted entirely in the Kichwa language. Those sessions are dedicated to teach both the language itself, as well as socio-cultural studies. According to the instructors, the three Kichwa participants successfully participated in those sessions, working in close collaboration with other fluent Kichwa speakers. The second and third authors have worked with these participants, and believe the three of them have achieved advanced oral comprehension skills because they are originally from the Mid Napo River. As indicated in §2.2, in those villages, the generations of parents and grandparents still speak the language on a daily basis; thus, the conditions to relearn Kichwa are relatively favorable.

### **6.3 Oral production**

To collect oral production data, we employed a similar strategy to the one used with the Kukamas. The participants organized the sets of cards and briefly described them creating a story.

The results for the oral production task are given in Figure 4. They show that participant kich-3 is at a more advanced level in the proficiency continuum compared to the other two participants, particularly with respect to the length of the stories and the overall number of words produced. kich-3's score for word types ( = 144) is above the average for the three participants (avg = 92). The only score in in which kich-1 and kich-2 are above kich-3 is speech rate. However, a chi-square test comparing word types and speech rate reveals that there is not a significant difference among the participants.<sup>7</sup> The assessment provided by the Kichwa specialist aligns well with most of the scores in Figure 4. An analysis of specific linguistic features produced by the participants is found in Figure 4.

### **6.4 Qualitative analysis**

### **6.4.1 Phonetics and phonology**

The participants kich-1, kich-2 and kich-3 do not differ much from fluent Kichwa speakers with respect to the production of different sounds, intonation and even accent patterns. One of the phonetic characteristics of Kichwa is the sonorization of the voiceless stops /p, t, k/ which become [b, d, g], respectively, after a nasal consonant. For example, /ñampi/ 'path' is realized as [ñambi], /inti/ 'sun' is produced as [indi]; /chunka/ 'ten' is produced as [chunga]. All the

<sup>7</sup> 2 : 4.4693, : 0.346208. The result for *word types* and *speech rate* is not significant at < 0.05. It becomes significant if we add *number of words*.

Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

Figure 4: Kichwas' oral production

participants apply this voicing process. The stress pattern in Kichwa falls on the penultimate syllable, but in the spontaneous speech of fluent speakers, a word can undergo reduction of the stressed vowel, in which case the accent falls on the last syllable. For example, /manáchu/ 'is not true' becomes [manchú]. This vowel reduction phenomenon is documented in the speech of these heritage speakers as well. A related phenomenon is the fact participant kich-3 stresses the last syllable of some words like [tupán] that are generally produced as /túpan/ 'to meet someone', although this issue seems marginal.

### **6.4.2 Morphosyntax**

Kichwa is an agglutinating, suffixal language; nominal and verbal words are constituted of a verbal and nominal root and their respective suffixes. Grammatical relations are expressed via case marking including the accusative *-ta*, and the indirect object *-ta* or*-ma*. Non-core arguments are marked by a set of postpositions, the most frequent being the locative *-pi*, the allative *-ma* (when the noun is nonhuman), the allative *-pam* (when the noun is human), the ablative *-manta*, and the comitative/instrumental *-wa*. The main predicate of the clause is marked mainly with person indexes including: *-sha* '1sg.fut', *-nki* '2sg', *-nka* '3sg.fut'. The main verb can also take the tense marker *-rka* 'past', aspectual markers such as *-ra/-hu* 'durative', *-shka/-ska* 'perfective', and the causative marker *-chi*. All these morphemes are basic and frequent in everyday conversation. The preferred order of

### 7 Indigenous languages in higher education: Case studies from the Amazon

constituents in the clause is SOV, but this pattern is flexible because the core and non-core arguments are morphologically marked, with the exception of the subject argument. Thus, moving arguments around the clause does not alter the propositional meaning of the utterance, although it may change its pragmatics (Papa Coquinche & Rosales Alvarado 2015).

The three Kichwa participants seem to know most of the morphemes listed above. Participants kich-1 and kich-2, however, are in the process of strengthening the proper use of these suffixes. Note that the presence or absence of these morphemes can change the meaning of an expression in substantial ways. The example in (6a) was extracted from the story produced by kich-1. It shows that this participant is learning to use the accusative -*ta*, the causative -*chi*, and the locative -*pi.* In the context of the story, (6a) is trying to make reference to the fact that the owner of the dog makes his pet happy. This is also inferred from the transitivizer -*ya* in the verb. But this example lacks the accusative marker -*ta* in *dog*, and the causative –*chi* in the verb. The target construction provided by the Kichwa specialist is given in (6b):

	- b. Pay-pa 3sg.pre-gen allku-*ta* dog-acc sumak-ta beautiful-advzr kushi-ya-*chi*-shka be.happy-trs-cau-pfv 'He made his dog very happy'

The following example was also produced by kich-1. It shows that the verb has the necessary morphology of a finite verb. However, this participant used the instrumental marker -*wa* instead of the locative marker -*pi* to indicate that the bench is where the sitting takes place. The target construction is given in (7b).

	- b. Chaymanda then chay dem runa person shuk one banka-*pi* bench-loc tiya-ri-rka sit-inc-pas 'Then that person sat on a bench'

Participants kich-1 and kich-2 also show some inconsistencies with respect to preferred order of constituents. The speech sample of kich-3 also shows some of

### Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

the same inconsistencies described above, but these are less frequent and mostly in the context of complex sentences.

One of the most salient morphological features of this language is the use of personal pronouns with the genitive marker -*pa* to express possession within the noun phrase. For example, in [*ñuka-pa yaya*] 'my dad', the first-person pronoun *ñuka* is marked by the genitive marker -*pa*. In spontaneous speech, however, fluent speakers tend to drop the genitive marker, and possession is expressed by word order alone. As a result, fluent speakers have two variant constructions for adnominal possession: [PRO-pa N] and [PRO N]. Participants kich-1 and kich-2 are able to use the first variant, with the genitive marker. However, kich-3 already uses both variants, as shown in (8) and (9). Note that, in (8), the suffix -*pa* is missing in [*pay wasima*] 'at his house'. Thus, kich-3 seems not only aware of the morphosyntactic variants but uses both of them effectively, which contributes to propel him towards the more advanced end of the proficiency continuum.


To form complex sentences, Kichwa employs subordinator suffixes, such as -*pi* to express 'temporal overlap', -*sha*/-*sa* to express 'manner', and -*nkapa* to express the purpose of the action conveyed in the main clause. kich-3 uses all these complex structures, as shown in the following extract from a story produced by this participant. Subordinators are employed in (10b), (10c), and (10d). Participants kich-1 and kich-2 use mostly sequences of simple clauses, although they tend to join them using discourse connectors, as shown in (11), below.

	- b. "maska-kri-sha look.for-go-1sg.fut wayu-kuna-ta fruit-pl-acc kan-kuna-ta 2sg-pl-ben kara-*nkapa*." give-pur
	- c. chasna like.this rima-*pi*, speak-temp pay-pa 3sg-gen mama mother puri-*sa* go.around-man sacha-man jungle-all ri-n, go-3sg

### 7 Indigenous languages in higher education: Case studies from the Amazon

d. sacha-pi jungle-loc puri-*pi* go.around-temp tupa-n find-3sg mishki sweet muyu-kuna-ta seed-pl-acc '(a) Mother speaks to her sons. (b) "I'll go look for fruits in order to give you (to feed you)". (c) Speaking like this, mother goes walking around towards the jungle. (d) When she was walking around in the jungle, she found sweet seeds'

### **6.4.3 Discourse pragmatics**

At the discourse level, there is a tendency to use connectors to link ideas conveyed in simple structures. This is particularly evident in the speech of kich-1 and kich 2. The extract below from a story by kich-1 shows this. The frequent use of *chaymanda* 'then' and adverbs of time such as *washa* 'then, later' is noticeable, as in (11a) and (11d). This speaker seems to be using connectors instead of subordinators to link clauses.

	- b. chay dem allku dog mana neg muna-rka want-pas
	- c. *chaymanda* then chay dem runa person wan–chi-shka die-cau-pfv chay dem lumucha-ta agouti-acc
	- d. wan-chi-shka die-cau-pfv *washa* after pay-pa 3sg-gen wasi-ma house-all ri-rka go-pas '(a) His owner, the kid, wanted to cure him [the dog]. (b) that dog didn't want to. (c) then that person killed the agouti. (d)After killing, he went to his house.'

An important feature of Kichwa is the focalizer -*ka*. This marker operates at the level of discourse to explicitly highlight a piece of information about which one wants to draw attention. Focalization strategies are an interesting aspect in the learning process of heritage speakers. The only participant that makes use of this suffix is kich-3; however, he makes an excessive use of the focalizer, as evidenced in (12) and (13). In sentence (12), three elements are focalized, in (13) four elements. Fluid, native speakers focus only one element per sentence.

Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado


Participants kich-1 and kich-2 do not employ the focalizer marker. In sum, some areas of pragmatics pose significant challenges for heritage learners of both Kichwa and Kukama.

### **7 Implications for instruction of heritage languages in Amazonia**

The aim of this study was to explore how much heritage speakers can achieved in the process of relearning their ancestral languages in higher education programs. An examination of the use of Kukama and Kichwa as a response to audiovisual stimuli by eight FORMABIAP alumni reveals that heritage languages can be relearned in the context of formal education, as long as other forces and context dynamics are present as well. Overall, all the Kukama and Kichwa heritage speakers in this study demonstrated strong comprehension skills and have also achieved varying degrees of production skills. Speakers show some patterns from beginners, to intermediate levels, to quite advanced proficiency.

In addition, this study found that some features found in the speech of heritage speakers are similar to those found in the speech of elder speakers, but there are also some distinctive emerging patterns. Some of them are listed in Table 4.

The findings in Table 4 are hardly surprising, as accelerated language change is expected in contexts of language endangerment and language revitalization (Hinton 2001, Vallejos 2016b). For example, speakers of Navajo (Athabaskan), recognize themselves as either traditionalists and non-traditionalists. The use of Navajo specialized vocabulary, as well as some structural features such as the hierarchy of classification of nouns (Hale 1973), is very rare among nontraditional speakers (Wayne et al. 2003). Another example comes from Blackfoot


Table 4: Some patterns identified in the speech of heritage speakers

### Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

(Algonquian). There are two varieties referred to as Old or High Blackfoot, spoken by elders, and New or Modern Blackfoot, spoken by the new generations. Among the features found in High Blackfoot is the extensive use of incorporation; however, this pattern is no longer used by the Modern Blackfoot speakers (Miyashita & Shoe 2009). In the Amazon, Danielsen & Terhart (submitted) report several structural innovations among latent-speakers of Baure (Arawakan). Thus, it is important to document the progress made by these learners, as well as the innovations that might be emerging in their speech.

Given that languages are surrounded by variation and diversity, we do not believe language instruction should be restricted to achieve an abstract ideal of standard. Yet the findings of this study point to some areas that may need attention in terms of curricula development. Providing specific suggestions for heritage language instruction is beyond this paper, but the analysis provided here could be taken as a starting point for Kukama and Kichwa. For example, given the central role of orality in the life and social organization of the communities, facilitating greater exposure and practice of storytelling could help the progress of heritage learners. In addition, practical activities, such as the description of specific processes and concrete objects, activities with visual stimuli such as those used here, could help promote oral production and the expansion of vocabulary. The design of grammar instruction sessions aimed at overcoming difficulties with specific patterns, for example some of those listed in Table 4, seems desirable. Finally, the need to build a community of practice cannot be underestimated. Real interaction outside the classroom with those who speak the heritage languages is key to progress in the learning process.

Another important finding of this study is the positive attitudes shown by all the participants towards their identity and their heritage languages. To relearn a heritage language, particularly one with limited communicative functionality and prestige, motivation and commitment seem crucial. In addition, documenting progress is also important. Setting achievable goals, developing autonomy, and continuous self-assessment of learning are critical.

The assessment of heritage language learners should go way beyond language skills and explicit grammatical knowledge. A holistic approach must include also pragmatic knowledge, including the control of socio-cultural language norms, as well as awareness of linguistic variation in their communities. As indicated above, language attitudes and ideologies surrounding these languages, as well as learners' cultural connection with their ethnic group should be part of the picture as well. Finally, their political positioning with respect to their communities and indigenous organizations is also essential, especially in the context of the Peruvian Amazon.

7 Indigenous languages in higher education: Case studies from the Amazon

### **8 Closing remarks**

Language revitalization initiatives have been taking place in Latin American for some time now; however, the impact of such efforts in the creation of new speakers has not been documented thoroughly. There is a need to establish the baseline for each heritage language. This task requires detailed knowledge of each specific language, the sociocultural context, the demographic patterns of dialectal variation, etc. For Amazonian languages, this information is generally not available or systematized, and would need to be collected prior to a serious assessment of specific revitalization initiatives.

Given the degree of endangerment of the heritage languages involved, especially for Kukama and for some Kichwa villages, a significant finding of this study is that endangered languages can be relearned in well-structured instructional settings. We demonstrate that the outcomes of the methodology implemented by FORMABIAP are both tangible and substantial. The results reported here can help advance the discussion about model design and strategies to work with heritage languages within the classroom and beyond. One of the most important findings is the participants' commitment to their language, their community, and their indigenous organizations. They express a strong motivation to continue to learn their heritage language as they teach it to children. It should be clear by now that Kukama and Kichwa are not conventional second languages and should not be treated as such in higher education. What pedagogical challenges heritage learners of indigenous languages bring to the classroom is an important question, and we hope to have contributed to the start of this discussion with this study.

### **Acknowledgments**

Our deepest gratitude goes to the eight Kukamas and Kichwas that have participated in this study. We gratefully acknowledge the contributions of Rosa Amías Murayari and Claudio Papa Coquinche, the Kukama and Kichwa specialists, respectively, who have helped us with the analysis of the data. This paper has benefited greatly from comments and discussions with Richard Ricopa Yaicate and Juan Manuel Vásquez Murayari, instructors of FORMABIAP, as well as three anonymous reviewers and the editors of the volume. Institutional support from Never Tuesta, the Coordinator of FORMABIAP, is also gratefully acknowledged. All errors and shortcomings are our own.

Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado

### **Abbreviations**


### **References**


7 Indigenous languages in higher education: Case studies from the Amazon


Rosa Vallejos Yopán, Fernando García Rivera & Haydée Rosales Alvarado


## **Chapter 8**

## **Pedagogical implications of assessment in multilingual contexts**

### Rosita L. Rivera<sup>a</sup> & Eva Rodríguez-González<sup>b</sup>

<sup>a</sup>University of Puerto Rico-Mayagüez <sup>b</sup>University of New Mexico

This chapter discusses the case of multilingualism and context-based approaches to assessment as an eclectic approach that requires a robust knowledge and understanding of the linguistic diversity of language learners. It also calls for a more inclusive approach in the design and implementation of assessment in higher education. We also connect the findings from the contributions to this volume and their research to an ecological approach to assessment as it relates to linguistic sensitivity in multilingual contexts. The chapter ends by suggesting future directions for research based on alternative assessment and connecting those to current issues in language learning and assessment.

> Assessment works when we learn to look at it as a process for improving the quality of our teaching. It works when we dialogue with colleagues, both within our discipline and across campus, and create new ideas to help students learn. Assessment works when we try something new and don't get disheartened when it doesn't work; instead, we reevaluate and try something else. Assessment works when something new proves effective and we gain information that moves our curriculum forward. Assessment can work if we quit making excuses as to why it's so difficult and messy and instead look to the information to reinforce what works and discard what doesn't. Assessment works when we embrace the challenge of always getting better

> > Vickie Kelly (2017, Washburn University)

Rosita L. Rivera & Eva Rodríguez-González. 2022. Pedagogical implications of assessment in multilingual contexts. In Eva Rodríguez-González & Rosita L. Rivera (eds.), *Language assessment in multilingual settings: Innovative practices across formal and informal environments*, 193–205. Berlin: Language Science Press. DOI: 10.5281/zenodo. 6762284

### Rosita L. Rivera & Eva Rodríguez-González

### **1 Linguistic sensitivity and pedagogical training in language assessment**

As researchers continue to develop multiple ways to document learning growth and development, the different needs for teacher training continue to grow. Language program coordinators designing curricula for language courses may face many challenges if they choose to focus primarily on proficiency skills and standardized language testing. For instance, in classroom-based language scenarios, interpersonal and intercultural competence may be left behind or not prioritized in terms of learning outcomes and, as a result, the social aspect of language development might not be monitored or even paid that much attention for assessment purposes. In this regard, Malovrh & Menke (2021) state that both the limitation and the challenge "is not necessarily in identifying the problem, but in allocating institutional resources (both human and financial) to revising and designing curricular sequences that systematically develop desired learning outcomes" (2021: 500). In this regard, Phakiti & Isaacs (2021) highlight the importance of assessment literacy and make a call to the scholarly community to empower teachers "to deal and communicate with external mandates such as government or state agencies who often impose external assessments on students and educational systems. For teachers, understanding assessment quality is more important than ever" (2021: 19). The authors recommend excellent resources for teachers (2021: Appendix A) and call them to be critical consumers of learning materials and assessment instruments (Brown & Trace 2017), including through professional development activities (Harding & Kremmel 2017).

The integration and emphasis on linguistically responsive instruction that is both inclusive and offers a variety of opportunities for activities and assessment tools should be explicit and prioritized in language learning contexts. However, as Huang & Laskowski (2014) point out for English second language teaching, translating such a view of language education into classroom practice requires the instructor to be linguistically sensitive to both the content and tasks that learners face during their own learning path. While studies exist to show how effective language instructors integrate language and content and prepare future language speakers for the job market, research attention is still emerging on how instructors are trained to be equipped with the needed knowledge and skills on assessment techniques. Thus, it is important to consider how instructortraining may benefit from integrating awareness of multilingual learners' realities. The notion of an "ideal learner" or "test-taker" could be challenged through instructor-training that presents different ways of assessing language learning

### 8 Pedagogical implications of assessment in multilingual contexts

to encourage collaboration and linguistic mediation among learners. When referring to training where teachers also learn about language anxiety, the role of emotions in the language classroom, power dynamics, and agency in and outside of the classroom is key to assessment of languages. In this regard, the chapters by Thompson (2022 [this volume]) and Rodríguez-González et al. (2022 [this volume]) in the present volume offer suggestions on the kind of activities and assessments that are sensitive to the multiple learner profiles within a given classroom setting. Thompson's IPAs and language portfolios are examples of multidimension assessment tools that measure language performance and consider learner reflection as an ongoing process that documents language growth. Similary, the chapters by Rodríguez-González et al. (2022 [this volume]) and Dickinson & Martinez (2022 [this volume]) in this volume identify a specific learner survey (Can-Do statements survey) that allows both the learner and instructor to monitor learner self-efficacy and language development.

Because assessment is process-oriented, it has significant potential for exploring language and culture from an interdisciplinary and midway perspective. By including multiple modalities of assessment such as those done via self-assessment and questionnaires where learners and community members share perceptions of their own language use, experience and learning such as the ones used in this volume by Silva in her article on Brazilian Heritage Portuguese and Vallejos et al. when assessing Kukama and Kichwa, we acknowledge challenges in learning and embrace innovative instructional practices in response to cultural and linguistic diversity. Additionally, by involving peers and community members in assessment practices, we will be allowing spaces for creative interpretation that include individual and collective voices that engage with each other when monitoring language development and personal growth. Through critical, selfreflexive practices embedded in our research about language learning, teaching and assessment, we can work against racial, cultural, linguistic, and socioeconomic inequalities by creating humane classrooms and/or communities of practice where learners and instructors learn together to use language and literacy in critical and empowering ways.

When preparing and training future language instructors or accreditors that will be in charge of documenting and monitoring language assessment, educators should serve as advocates and models of social justice and equity. Social justiceoriented instructors and trainers play a significant role in seeking alternative ways to address various forms of official knowledge with the learner populations they serve, especially forms of official knowledge that marginalize certain groups while privileging others. For instance, language assessments should be shaped according to multiple heritage language profiles and second language learners.

### Rosita L. Rivera & Eva Rodríguez-González

In this volume, Rodríguez-González et al. (2022 [this volume]) describes assessment tools and provides instructions for implementation in classroom settings for both heritage language speakers and second language learners. Additionally, the chapter by Hernández (2022 [this volume]) on pragmatic development offers suggestions for assessing pragmatic language growth in study abroad language settings. By means of reviewing the existing literature on the topic, Hernández carefully examines how pragmatic knowledge can be measured and included as another key area in language learning and curriculum design. Instructors should also envision classrooms as safe spaces for struggle and transformative action and social change. For instance, community-based language learning and study abroad experiences (see Hernández 2022 [this volume], Silva 2022 [this volume] and Dickinson & Martinez 2022 [this volume]) offer a plethora of opportunities for creative and reflective assessment practices that allow learners to shape their own transformations and interconnect themselves with others in a more naturalistic way than what a traditional classroom and curriculum might offer.

In a 2005 position statement from the National Council of Teachers of English (NCTE 2005) entitled "Supporting Linguistically and Culturally Diverse Learners in English Education", the following beliefs and recommendations were suggested as a call for action. All of them may serve well as a point of reference for consideration for instructors, program coordinators and scholars working on alternative language assessments that are sensitive to the multiple realities of learners: (1) respect for all learners; (2) funds of knowledge; (3) inquiring into practice; (4) variety of educational experience; (5) modeling practice; (6) critical users of language; (7) crossing cultural boundaries; and (8) teaching as a political act. These recommendations from the NCTE 2005 report, although more aligned for teaching per se, have implications for language assessment. For example, instead of creating language activities and assessments where accuracy in sentence formation and vocabulary use may be biased towards a variety or dialect of the target language as determined by power and prestige, instructors acknowledge and give credit for different ways of addressing people, different culture-driven practices to denote proximity and kindness as a way to respect all learners and speakers. How do instructors and teacher educators successfully integrate the funds of knowledge learners bring into their pedagogic and assessment stance? Reflection journals and presentations of community-based projects such as the ones identified by Thompson in the present volume open a myriad of opportunities for intercultural competence regardless of whether the learner has shared that reflection on the target language or not. The learner becomes his/her own agent and funds of knowledge for assessment. The assessment piece in learner's reflections would not need to have a rubric or a scale of points for a grade, but

### 8 Pedagogical implications of assessment in multilingual contexts

rather credit should be confirmed via a checklist of the items requested by the instructor to be included in the reflection (subjective scoring based on evidence provided in the reflection as checked by both learners and instructor). For instance, the Can-Do statements survey used by Rodríguez-González et al.'s and Dickinson & Martínez in this volume serves as an example of an assessment tool that uses reflection and is learner-driven.

Educators may benefit from learning more about sociolinguistics both in teacher preparation programs and in ongoing professional development. Developing this kind of knowledge may help to avoid language marginalization (Delpit & Kilgour Dowdy 2003). By training future instructors in alternative assessments, they will have a positive impact on the attitudes towards those assessment (vs. traditional ones) and their own attitudes and beliefs about the nature of language learning may organically evolve as well when applying multiple ways and tools of assessment (see Shahbari & Abu-Alhija 2018 for findings related to prospective Mathematics teachers on alternative assessments). Examples of professional development related to alternative assessments could include but not limited to workshops on the inclusion of portfolios as "bodies of evidence" to document learner's individual paths and growth in multiple communities of practice that differ substantially from the traditional classroom (see Thompson 2022 [this volume] for Portfolio recommendations and Green's (2014) PRICE principles for promoting effective classroom assessment- Planning, Reflection, Improvement, Cooperation, and Evidence).

### **2 An ecological approach to assessment**

As we finished this volume, we were in the middle of a pandemic that challenged language instruction and assessment in all contexts. Language practitioners are now faced with new approaches to dealing with a crisis. The pandemic is one example that challenged everyone, but there are also multiple contexts dealing with other issues such as natural disasters. Learners and educators may not have access to the same sources, which creates a disparity between formal and informal assessment (Malovrh & Menke 2021). These disparities are largely due to factors ranging from design of formal and informal assessment to measuring proficiency. Yet, learners' futures depend on the design and implementation of these instruments. Ortega (2017) and Mazak & Carroll (2016) make a case for challenging the ontological view of monolingual ideologies in language research and practice. New lines of inquiry are also undertaking an ecological orientation in language learning and assessment in SLA (Larsen-Freeman 2017). This epistemological view of a dynamic and contextual approach to assessment could also be

### Rosita L. Rivera & Eva Rodríguez-González

applied to foreign language instruction in multilingual contexts. This perspective acknowledges that language learning and assessment do not take place in isolation from the temporal space in which they occur, but they are dynamic in nature and change with the environment (Larsen-Freeman 2017). This perspective not only considers the traditional and individual factors such as age, motivation, aptitude and attitude, but also considers the underlying issues related to the learners as individuals in constant interaction with their historical and sociocultural, and sociopolitical contexts. Examples of these are issues related to learners' anxiety, emotions, values and beliefs about the language. In this volume, programs such IMPACT (see Dickinson & Martinez (2022 [this volume])) and the FORMABIAP (Vallejos Yopán et al. 2022 [this volume]) serve as a couple of examples of projects that include sociolinguistic profiles of learners for pedagogical and career-decision purposes. Such programs address the nature of the environment and the learners' willingness to communicate. Assessment within this kind of framework challenges traditional designs of instruments to measure learners' ability to communicate in the language. This requires seeing assessment and the learner's progress as a process rather than a product. It calls not for a one-time test to measure ability, but rather to study the needs of the students in a given context and their trajectory as language learners. It also includes validating the different linguistic repertoires they bring to the classrooms and provide experiences that will allow them to overcome the challenges they face in and outside of the classroom. As Larsen-Freeman (2017) explains, "languages are not only acquired or learned, but lived." (Ros i Solé 2016).

Assessment is also challenged by contexts in which monolingual ideologies continue to define what goes on in multilingual classrooms. Important pedagogical principles in an ecological approach are the creation of ecologically valid contexts, relationships, agency, motivation and identity. Some guidelines for applying an ecological perspective in language assessment may include (1) notion of "localness" (Freeman 2000, Tudor 2003), (2) contexts, (3) cultures of learning (Tudor 2003; Cortazzi & Jin 1996), and (4) teaching-learning dynamics (Tudor 2001). In this regard, Silva in this volume calls for including multiple voices and agents in the placement and teaching of Brazilian Heritage Portuguese Communityschools, an ecological pedagogical approach that should be designed after carefully examining the needs of the local community, the teaching approaches in practice and the dynamics between the learner, family and school respectively.

An ecological perspective also defines the classroom and assessment differently. While we have gained much knowledge about the purpose of assessment practices in the last decades, we have also become aware of challenges in applying traditional concepts of assessment to classroom-based assessment and other

### 8 Pedagogical implications of assessment in multilingual contexts

communities of practice outside the traditional classroom setting. Classroom assessments as the ones described in the present volume followed Turner's (2012) recommendations when involving the use of strategies by instructors to plan and carry out the collection of multiple types of information concerning student language use, the analysis and interpretation of data collected for assessment, the feedback received and how the information gathered helps make present and future decisions to enhance teaching and learning (2012: 65). As some of the chapters in the present volume included examples of classroom assessment techniques used in Higher Education in the US, those assessments must be regarded as unique to a given teaching context and, therefore, approaches to classroom assessment validity need to be dynamic, sociocultural in nature and different depending on the community of speakers and learners. The multiple alternative assessments proposed in the present volume may fit well under edumetric approaches to validity in assessment as they all aimed to promote and foster good learning behavior and successful progress. Under these parameters, assessment processes become an essential part of everyday classroom practice and involve both instructors and learners in reflection, dialogue and decision making with the ultimate goal of using assessment *FOR* learning (AfL: Assessment Reform Group, Broadfoot et al. 2002, Leung & Rea-Dickins 2007).

Returning to the analogy posed in the Introduction of the present volume (Chapter 1), we view the dynamics and shaping of language and its related assessment as a continuous fractal formation similar to those fractals that are everywhere in nature. Ecologists have found fractal geometry to be an extremely useful tool for describing ecological systems. Population, community, ecosystems, and landscape ecologists use fractal geometry as a tool to help define and explain the systems in the world around us. The fractal dimension is conceived as a measure of the nature of habitats. In language contact and education settings, the habitats are communities of practice. Different tools are required in population ecology because the resolution or scale with which field data should be gathered is attuned to the study organism (individual learners in educational contexts). Insect movements and plant root growth follow a continuous dynamic path but the tools required to measure this continuous pathway are very different. Despite multiple shared characteristics of learners' profile, the assessment tools are unique and different depending on the habitat (e.g. classroom walls, communitybased learning). In order to avoid habitat fragmentation that produces isolated patches (minority language profiles such as heritage language learners for instance), fractal formation and assessment need to be accessible and inclusive to all in an equitable manner and should keep evolving in varied patterns.

Rosita L. Rivera & Eva Rodríguez-González

### **3 Looking ahead: Future directions for research on alternative assessment in language learning**

### **3.1 Challenges of implementing inclusive approaches to assessment in higher education**

One of the strongest influences of assessment in higher education is validity theory (Phakiti & Isaacs 2021, Brown & Trace 2017). Both formal and informal assessments need to take into account validity and clear standards in order to elicit data to support learners and to design curriculum. Scholars have argued different dimensions of validity in assessment and the consequences for learners. McNamara & Roever (2006) argued that for the most part, language research investigates technical aspects of validity and not the social dimensions. In other words, assessment is designed for a given context and the impact on this particular community of practice should be considered within construct validity (the extend to which the assessment instrument is intended to measure), content validity (the inclusion of content that is within the scope of the course material covered in class), criterion related validity, and consequential validity (the intended or unintended consequences of assessment for the learners, for instance, being able to graduate or join a program).

Another issue is psychometric test validity when assessing learners in classrooms. As Phakiti & Isaacs (2021) argued, "classroom assessment scores cannot be correlated with other external test scores because classroom assessment is used to help the students to improve their skills and overcome any learning difficulties through instructional support" (2021: 9). Assessment quality should be the approach. In their model, the authors proposed the following components for what they called *assessment quality* rather than *validity.* This model includes the following components: (1) validity, (2) reliability, (3) practicality, (4) authenticity, (5) ethics, (6) fairness, and (7) effect. It also proposes to start with the intended learning outcomes, and that classroom activities and classroom assessment need to be aligned with the outcomes.

Yet, the challenges classroom instructors continue to face are related to institutional policies as well as the rise in standard tests assessment. This generates in some cases the need for teachers to focus on the tests rather than on the learning process and improvement of their students. Another issue is that alternative forms of assessment require more time spent in the planning stages of the activities. The development of some assessment instruments such as rubrics and group activities to assess different language skills may also require cooperation between teachers. Depending on the class composition, there will be a variety of

### 8 Pedagogical implications of assessment in multilingual contexts

challenges related to group work and task completion. However, these activities are more aligned with providing learners with opportunities to engage with their context. Awareness and understanding of assessment practices, institutional policies, and students' academic needs as well as their sociocultural context should be considered in the design of assessment that fosters students' learning through assessment quality criteria.

### **3.2 Computer-mediated communication and assessment in multilingual contexts**

The role of technology in assessment has evolved over the last two decades. The language learner of the 21st Century is constantly learning through different computer-mediated communication outlets. Social media technology and our ability to text and communicate in different languages has revolutionized and expanded the way we communicate and use languages (Thorne et al. 2021; Crystal 2009; Thorne 2008) and how we assess language learners through the use of videogames (Thorne et al. 2012; Gee 2014). As global citizens, we have moved quickly from texting with limited characters and communicating a message through short texts, to smart phones and social media in which we can basically generate entire documents to share with a wide audience through multimodal approaches to literacy. This has also facilitated processing information in different ways and through multiple languages including the use of technology and *netspeak* (Crystal 2009). These technologies have challenged our reality and altered how we communicate in real time with others around the world. Learners are now more interested in learning foreign languages due to their particular interests. Watching *anime* cartoons, reading comic books such as *Manga* for Japanese learning, and playing videogames, are examples of the motivations for younger generations to use language. Assessment practices are also challenged by these new ways of learning. As such, in multilingual contexts learners are tested in two or more languages simultaneously and their performance is assessed on how well they complete a given task instead of how well they validate and use one of their languages. In this regard, Larsen-Freeman (2018) fittingly points out that "while this type of assessment may not be widely adopted any time soon, computer adaptive testing may well lend itself to more developmentally sensitive, self-referenced assessment, instead of approaches that resemble traditional standardized exams" (2018: 63).

Recent events such as the pandemic and natural disasters are forcing language educators to rethink assessment during these challenging times. Alternative assessments become the way in which the world continues to collaborate through

synchronous and asynchronous instruction. This has also generated new opportunities to engage with learners in different ways that were not previously considered by more traditional assessment methods.

The present volume aimed to provide a snapshot of some alternative assessments that address different realities and needs and initiate a dialogue on future research and additional modalities of alternative assessments in language learning in different communities of practice with other learner profiles. The limited scope of the research findings covered in this volume was due to the contexts the authors were operating mostly in Higher Education in the United States, and the Amazon of Peru and the languages involved (e.g. Heritage language learners of Spanish, Portuguese). Further research in language assessment should include additional language profiles in multiple communities of practice such as sign language learners, heritage language learners of other languages different from Spanish in the U.S. for instance, and study of language assessment in indigenous communities around the globe, to identify a few. Another area of much needed research would be assessment of language learning in hybrid and fully online educational settings. Additionally, pedagogical and assessment challenges (and solutions) remain to be fully explored when addressing disparities between formal and informal assessment practices, validity issues and teaching training. An interesting line of inquiry worth pursuing when advancing our knowledge on alternative assessments for language development would be to determine learners' (and also instructors') dispositions by assessing how receptive they are to the proposed assessments, how willing they are to continue to learn about them, apply them, and be influenced by them. The research findings presented in this volume, though yielding more questions than answers, provides a promising research agenda and dialogue for scholars interested in assessment of language learning.

### **References**


8 Pedagogical implications of assessment in multilingual contexts



8 Pedagogical implications of assessment in multilingual contexts


Abeywickrama, Priyanvada, 131 Abu-Alhija, Fadia Nasser, 197 Adair-Hauck, Bonnie, 20–22 Aida, Yukie, 37, 39 Akar, Tahmina, 23 Alam, Md. Jahangir, 23 Alarcón, Irma, 167 Alcón-Soler, Eva, 75, 79, 91 Amías, Rosa, 166, 167 Andersen, Roger W., 83 Anderson, Aaron, 78, 81 Ash, Sarah L., 18 Bachman, Lyle F., 67 Back, Michele, 78 Bailey, Alison L., 131, 132 Bandura, Albert, 37, 38 Bardovi-Harlig, Kathleen, 75, 79, 94 Barron, Anne, 67, 68, 75–78, 81, 82, 94 Bastos, Maria-Thereza, 75, 79, 94 Bataller, Rebeca, 68, 69, 71, 72, 78, 80, 81, 85 Batalova, Jeanne, 135 Bates, Douglas, 117 Beaudrie, Sara M., 59, 105, 107, 128, 130, 131, 141–143 Beckett, Gulbahar H., 144 Bell, Nancy D., 76, 84 Bennett, Milton J., 12 Berbery, Maria Luz, 112, 118 Bernhardt, Elizabeth B., 12

Bert, Michel, 155, 157 Betz, Nancy E., 108, 115 Blizzard, Brittany, 135 Block, David, 140 Blum-Kulka, Shoshana, 80 Blumenfeld, Phyllis C., 144 Bodmer, Richard, 156 Boero, Paulo, 68, 69, 73, 74, 85–87, 89, 92, 94 Boruchowski, Ivian Destro, 133, 134, 140 Bouton, Lawrence F., 86, 87 Bradunas, Elena, 133, 142 Brand, Betsy, 106 Bringle, Robert G., 14 Broadfoot, Patricia, 199 Brown, Alan V., 3, 13, 18 Brown, Henry Douglas, 5, 131 Brown, James Dean, 194, 200 Brown, Penelope, 80, 89 Buriel, Raymond, 105, 112 Byram, Michael, 12 Byrnes, Heidi, 12 Carpi, Anthony, 112 Carreira, Maria M., 59, 107, 127, 130, 131, 134, 142, 143 Carroll, Kevin S., 197 Carvalho, Ana M., 141 Castellano, Marisa, 107 Center for Applied Linguistcs, 129 Chappuis, Jan, 13

Cheng, Stephanie Weijung, 77 Cheng, Yuh-show, 37, 39 Clark, M. Carolyn, 40 Clark-Gareca, Beth, 130–132, 142, 144 Clayton, Patti H., 18 Clifford, Joan, 14, 19 Cohen, Andrew D., 67–74, 76, 78, 80, 81, 85–87, 93, 94 Cole, Simon, 78, 81 Collett, Paul, 36 Compton, Carol J., 133 Condon, William, 23 Cordella, Marisa, 75, 76, 83 Correa, Maite, 59 Cortazzi, Martin, 198 Coryell, Joellen E., 40 Cox, Troy L., 36, 59 Crystal, David, 201 Culpeper, Jonathan, 72 Cummins, Jim, 59 Cuza, Alejandro, 78, 79 Czerwionka, Lori, 78, 79 Dahl, Merete, 72, 75 Daly, John A., 39 Danielsen, Swintha, 186 Darra, Maria, 36 De Pablos Ortega, Carlos, 77 Delett, Jennifer S., 26 Delgado-Romero, Edward A., 106 Delpit, Lisa, 197 Dewaele, Jean-Marc, 38, 61 Dias de Quadros, Magaly, 140 Diaz Maggioli, Gabriel, 20 DiBartolomeo, Megan, 76, 86, 87, 90 Dickinson, Kendra V., 195, 196, 198 Dings, Abby, 72, 75, 76, 84 Dorian, Nancy C., 157

Douglas, Masako O., 133, 134, 143 Ducar, Cynthia, 130, 141, 142 Dufon, Margaret Ann, 69, 72, 74, 75 Dwyer, James, 106 Edwards, Bob, 14 Elder, Catherine, 132 Ellis, Rod, 39 Fairclough, Marta, 107, 127, 128, 130, 141, 142 Félix-Brasdefer, J. César, 68, 69, 71– 78, 80, 82, 83, 92–94 Foster, Charles R., 140 Fox, Janna, 23 Freeman, Donald, 198 Fukasawa, Emi, 77 Ganschow, Leonore, 61 García, Fernando, 154 García, Ofelia, 1 Gardner, Robert C., 16 Gee, James Paul, 201 Genesee, Fred, 23 Godoy, Cíntia, 140 Golato, Andrea, 69 Gómez Rodrígues, Luis Fernando, 144 Gonzales, John M., 106 Gonzalez, Virginia, 133 Gordon, Leslie, 16, 17 Green, Anthony, 197 Green, Nancy, 128 Grinevald, Colette, 155, 157 Gushue, George V., 112, 115 Gutierrez, Roberto, 134 Hale, Ken, 184 Halenko, Nicola, 69, 72, 73, 85–88 Hamp-Lyons, Liz, 23

Han, Sangkyung, 78, 80, 81 Harding, Luke, 194 Hasler-Barker, Maria, 68, 69, 71, 72, 74, 75, 77, 83, 93, 94 Hassall, Tim, 75, 84 Hedgcock, John S., 40, 41 Henery, Ashlie, 86, 87 Hernández, Todd A., 67–69, 72–74, 76, 78, 80, 82, 85–87, 89, 90, 92–94, 196 Hinton, Leanne, 161, 184 Hlas, Anne Cummings, 36 Hoffman-Hicks, Sheila D., 77, 80 Horwitz, Elaine K., 38, 39 Huang, Jingzi, 194 Ilieva, Gabriela N., 130–132, 142, 144 Isaacs, Talia, 194, 200 Ishida, Midori, 76, 84 Iwasaki, Noriko, 76, 83 Jacoby, Barbara, 14 Jin, Lixian, 198 Jocson, Korina M., 107 Johnson, Steven, 12 Jones, Christian, 85, 87, 88 Kagan, Olga, 161, 172 Kasper, Gabriele, 67, 70–72, 75 Kelley, Colleen, 84 Kilgour Dowdy, Joanne, 197 Kim, Sung-Yeon, 38, 39 Kinginger, Celeste, 75–77, 79, 84 Klee, Carol A., 13, 23, 24, 26 Klein, Larissa, 36 Kondo, Sachiko, 76, 80 Kondo-Brown, Kimi, 135 Krashen, Stephen D., 55 Kremmel, Benjamin, 194

Kunschak, Claudia, 26, 27 Lam, Ricky, 23, 24 Lamboy, Edwin, 17 Lantolf, James P., 1 Larsen-Freeman, Diane, 1, 3, 197, 198, 201 Laskowski, Terry, 194 Leeman, Jennifer, 128 Lefkowitz, Natalie, 40, 41 Leung, Constant, 199 Levinson, Stephen C., 80, 89 Li, Shuai, 69, 73, 78, 82, 93 Lico, Ana Lúcia, 140 Lira, Camila, 140 Litran, Christina, 140 Little, David, 59 Liu, Na, 133, 143 LoCastro, Virginia, 68 Long, Michael Hugh, 42 Lopez Odango, Emerson, 161 López, Luis Enrique, 154 Lowther Pereira, Kelly, 17, 18 Lu, Tommy, 129 Lucero, Audrey, 133 Luzzo, Darrell Anthony, 112 MacGregor-Mendoza, Patricia, 130, 141 MacIntyre, Peter D., 38, 39, 41 Magnan, Sally Sieloff, 12, 78 Malovrh, Paul A., 1, 5, 194, 197 Márquez Reiter, Rosina, 82 Martin, Cynthia, 165 Martinez, Glenn, 195, 196, 198 Martínez-Flor, Alicia, 85 Masuda, Kyoko, 75, 76, 84 Matsumura, Shoichi, 77, 79, 94 Mayor, Pedro, 156

Mazak, Catherine M., 197 McKinnon, Sean, 76 McMillan, James, 23–25 McNamara, Tim, 200 McTighe, Jay, 134 McWhirter, Ellen Hawley, 112 Medina, Almitra, 16, 17 Mejia-Smith, Brenda, 112 Menke, Mandy R., 1, 5, 194, 197 Meyers, Judith, 84 Miller, Michael D., 39 Miller, Paul Chamness, 144 Mills, Nicole, 44 Mir, Montserrat, 87, 91 Miyashita, Mizuki, 186 Moeller, Aleidine K., 36, 37, 42, 45, 165 Montrul, Silvina, 59, 161 Mooney, Linda A., 14 Morris, Kimberly J., 69, 85–87, 89 Moseley, Christopher, 155 Norris, John M., 12, 15, 42 Nunan, David, 38 O'Brien, Karen M., 112, 118 O'Malley, J. Michael, 13, 20 Ojeda, Lizette, 112 Ortega, Lourdes, 1, 3, 5, 197 Osuka, Naoko, 75, 79 Palmer, Adrian S., 67 Papa Coquinche, Claudio, 181 Papanthymou, Anastasia, 36 Parodi, Claudia, 161 Parra, Maria Luisa, 107 Parrot, 112 Pedersen, Paula J., 12

93 Phakiti, Aek, 194, 200 Phillips, Elaine M., 37–39 Pierce, Lorraine Valdez, 13, 20 Pinto, Derrin, 82 Polinsky, Maria, 161, 172 Potowski, Kim, 107, 161 Queixalós, Francesc, 154 Raoofi, Saeid, 38 Rea-Dickins, Pauline, 199 Reisinger, Deborah S., 14, 19 Ren, Wei, 68, 69, 72, 73, 75, 78, 82, 83, 93 Rifkin, Benjamin, 12 Rockey, Catherine, 72, 74, 93 Rodríguez, Silvia, 78, 79 Rodríguez-González, Eva, 195, 196 Roever, Carsten, 69, 70, 79, 200 Ros i Solé, Cristina, 198 Rosales Alvarado, Haydée, 181 Rose, Kenneth R., 67, 70 Rothman, Jason, 161 Russell, Victoria, 87, 90 Salgado-Robles, Francisco, 17 Sánchez Hernández, Ariadna, 75, 79 Sanz, Cristina, 68 Schauer, Gila A., 69, 72, 76, 78–80 Schmidt, Richard, 85 Schölmberger, Ursula, 76, 80, 81 Schulz, R. A., 12 Shahbari, Juhaina Awawdeh, 197 Shamo, Wayne G., 39 Shapiro, Samuel Sanford, 121 Shively, Rachel L., 67–76, 78, 80, 81, 83–88, 92–94

Pérez Vidal, Carmen, 68, 83, 85, 92,

Perclova, Radka, 59

Shoe, Shirlee Crow, 186 Silva, Gláucia V., 134, 196 Silva-Corvalán, Carmen, 165 Slavin, Robert, 134 Solís Fonseca, Gustavo, 154 Son, Young-A, 130, 141 Souza, Ana Luiza de, 140 Sparks, Richard L., 61 Stiggins, Rick, 13 Stipanovic, Natalie, 112 Sullivan, Kristen, 36 Swender, E., 12 Sykes, J., 12 Taguchi, Naoko, 68–70, 72–79, 82, 84, 94 Tallon, Michael, 40, 41 Taylor, Karen M., 108, 115 Tedick, Diane J., 13, 23, 24, 26 Teng, Chunhong, 87, 89 Terhart, Lena, 186 Thomas, Jenny, 68 Thompson, Gregory L., 13, 15, 18, 107, 195, 197 Thorne, Steven L., 201 Topping, Brett, 133, 142 Torres, Kelly Moore, 38, 40, 41 Trace, Jonathan, 194, 200 Trapnell Forero, Lucy, 160 Troyan, Francis J., 20–22 Tschirner, Erwin, 12 Tudor, Ian, 198 Turnbull, William, 71 Turner, Carolyn E., 199 Turner, Jeannine E., 38, 40, 41 Upshur, John A., 23 Usó-Juan, Esther, 85, 88

Vaca Torres, Ady Marcela, 144

Valdés, Guadalupe, 40, 44, 128, 130, 141, 161, 165, 171 Vallejos Yopán, Rosa, 198 Vallejos, Rosa, 155, 158, 166, 173, 174, 176, 177, 184 Van Lier, Leo, 3 Vande Berg, Michael, 68 Vásquez, Camilla, 87, 90 Vergara Wilson, Dámian, 107, 141 Vilar Beltrán, Elina, 78, 79 Wang, Shuhan C., 128 Warga, Muriel, 76, 80, 81 Wayne, Holm, 184 Wei, Li, 1 Weibell, Christian J., 37 Wenden, Anita J., 38 Wewer, Tania, 25, 26 Wiggins, Grant, 20, 134 Wilk, Martin B., 121 Winke, Paula M., 87, 89 Woodfield, Helen, 69, 73, 78 Woodrow, Lindy, 39 Young, Dolly J., 39 Yu, Fei, 37, 42, 45, 165 Ziegler, Janet D., 130 Zyzik, Eve, 59

## Language assessment in multilingual settings

This volume explores and addresses questions related to equitable access for assessment. It seeks to initiate a conversation among scholars about inclusive practices in language assessments. Whether the student is a second language learner, a heritage language learner, a multilingual language speaker, a community member, the authors in the present volume provide examples of assessment that do not follow a single universal or standardized design but an applicable one based on the needs and context of a given community. The contributors in this volume are scholars from different disciplines and contexts in Higher Education. They have created and proposed multiple lower-stakes assignments and accommodated learning by being flexible and open without assuming that learners know how to do specific tasks. Each chapter provides different examples on Justice, Equity, Diversity, and Inclusion (JEDI) assessment practices based on observation, examination, and integrative notions of diverse language scenarios. It may be of interest to researchers and practitioners in the fields of curriculum and instruction, language learning, and applied linguistics as well as those in the field of language teaching in general. Thus this volume broadens the scope of research in the area of multilingual assessment.